Question

You’ll create the parser with a 2-part layered approach. You’ll first create a Lexer, which will read characters from a file and output tokens. Then you’ll write the second part, which will process the tokens and determine if there are errors in the program.

This program have 3 files Lexer.java, Token.java, Parser.java

Part 1 Lexer:

The class Lexer should have the following methods:
public constructor which takes a file name as a parameter○private
getInput which reads the data from the file into a char array
public getNextToken which returns a Token object
private functions for getIdentifier and getInteger.
These helper functions should handle extracting the rest of an identifier/integer once one is identified
A main function that calls getNextToken within a loop and prints out all the tokens in the given file.

For example, if your file contains:

aa=34

bb=45+6

The Lexer will print:

TokenType Token Value

Identifier aa

AssignOp =

Integer 34

Identifier bb

AssignOp =

Integer 45

PlusOp +

Integer 6

EndOfFile -

Part 2 Parser

Define the class Parser with the following methods:
A constructor which creates a Lexer as a data member
parseProgram -- this method drives the process and parses an entire program.
Thismethod should call parseAssignment within a loop
parseAssignment -- this method should parse a single assignment statement. It shouldcall parseId, parseAssignmentOp, and parseExpression.
parseId-- this method parses a single identifier
parseAssignOp-- this method parses a single assignment operator
parseExpression-- this method parses an expression, i.e., the right-hand-side of anassignment. Note that expressions can include + signs, e.g., Y+3+4
a main method which just creates a Parser and calls ParseProgram.

For instance, the code:

x12=17+43

a = x12+35+1

is syntactically correct for the SIMPLE language, and the parser should just report, “valid program”.

The code: 17=x+3 is not valid and the parser should report, “Expecting Identifier on line 1”.

Starting code

import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.FileReader;

public class Lexer {

final int MAXSIZE=3000;
char buffer[]; // this will hold the chars of the file
int bufSize=0; // how many chars their actually are
int index=0; // the location within buffer as we scan

public Lexer(String fileName) {
this.buffer = new char[MAXSIZE];
getInput(fileName);

}
public void getInput(String fileName) {
File inputFile = new File (fileName);
try (FileReader reader = new FileReader(inputFile)) {
this.bufSize = reader.read(this.buffer,0,1000);
} catch(FileNotFoundException fnf) {
System.out.println("File not found.");
System.out.println(fnf.getMessage());
System.exit(1);
} catch(IOException exc) {
System.out.println("IO Exception");
System.out.println(exc.getMessage());
System.exit(1);
}

}
// just print out buffer, mainly for debugging
public String toString() {
int i=0;
String s="";
while (i s = s+this.buffer[i];
i++;
}
return s;
}
// this method scans the buffer starting at index. If it finds a lowercase letter, it
// continues to scan and adds the lower case letters to a string until a non lower-case letter
// is scanned. this.index is incremented and will end up on the last lowercase letter
// NOTE: THIS IS A SAMPLE METHOD YOU DO NOT NEED IN LEXER
public String firstWord() {
String s="";
char c = this.buffer[this.index];
while ((c>='a') && (c<='z')) {
s=s+c;
this.index++;
c = this.buffer[this.index];
}
return s;

}

public static void main(String[] args) {
Lexer lexer = new Lexer("token_file");
System.out.println(lexer);
System.out.println(lexer.firstWord());

}

}

Answer 1

Please let me know if you have any doubts or you want me to modify the answer. And if you find this answer useful then don't forget to rate my answer as thumps up. Thank you! :)

import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.FileReader;

public class Lexer {

    final int MAXSIZE = 3000;
    char buffer[];
    int bufferSize = 0;
    int index = 0;

    public Lexer(String fileName) {
        this.buffer = new char[MAXSIZE];
        getInput(fileName);

    }

    private void getInput(String fileName) {
        File inputFile = new File(fileName);
        try (FileReader reader = new FileReader(inputFile)) {
            this.bufferSize = reader.read(this.buffer, 0, 1000);
        } catch (FileNotFoundException fnf) {
            System.out.println("File not found.");
            System.out.println(fnf.getMessage());
            System.exit(1);
        } catch (IOException exc) {
            System.out.println("IO Exception");
            System.out.println(exc.getMessage());
            System.exit(1);
        }

    }

    public Token getNextToken() {

        if (buffer[index] == '\n') {
            index++;
            //return (new Token("\n", "\n"));
        }

        if (buffer[index] == ' ')
            index++;

        while (this.index < this.bufferSize || this.buffer[index] != '-') {

            if (Character.isDigit(this.buffer[index])) {

                return getInteger();

            } else if (Character.isLetterOrDigit(this.buffer[index])) {
                return getIdentifier();

            } else if (this.buffer[index] == '+') {
                index++;
                return (new Token("PlusOp", "+"));

            } else if (this.buffer[index] == '=') {
                index++;
                return (new Token("AssignOp", "="));

            } else if (this.buffer[index] == '#') {
                index++;
                return (new Token("Unknown", "#"));

            }else if (this.buffer[index] == '$'){
                index++;
                return (new Token("Unknown", "$"));

            } else if (this.buffer[index] == '-') {
                return (new Token("End of File", "-"));
            }

        }
        return getNextToken();

    }

    public String toString() {
        int i = 0;
        String s = "";
        while (i < this.bufferSize) {
            s = s + this.buffer[i];
            i++;
        }
        return s;
    }

    private Token getIdentifier() {
        String str = "";
        while ((this.index < this.bufferSize) && Character.isLetterOrDigit(this.buffer[index])) {
            str += this.buffer[index];
            index++;
        }
        return (new Token("Identifier", str));
    }

    private Token getInteger() {
        String str = "";

        while ((this.index < this.bufferSize) && Character.isDigit(this.buffer[index])) {
            str += this.buffer[index];
            index++;

        }
        return (new Token("Integer", str));
    }

    public static void main(String[] args) {
        Lexer lexer = new Lexer("/Users/swapnil/IdeaProjects/LexerParser/src/input.txt");
        //System.out.println(lexer);

        System.out.println("Token Type    Token Value\n");
        Token tok = lexer.getNextToken();
        System.out.println(tok);

        while (!tok.getType().equals("End of File")) {
            tok = lexer.getNextToken();
            System.out.println(tok);

        }

    }

}

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
public class Token {
    private String type;
    private String value;
    //private final EOF = '-';

    public Token (String type, String value){
        this.type = type;
        this.value = value;

    }

    public String getType() {
        return type;
    }

    public String toString(){
        return (type + "        " +value);
    }

    public boolean equals(Token other){
        return this.value == other.value;
    }

}

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
public class Parser {
    Lexer l1;
    Token tok;

    //tok.getType()

    public Parser(Lexer l1){
        this.l1 = l1;
        tok = l1.getNextToken();

    }

    public boolean parseExpression(){

        System.out.println(tok);

        while (!tok.getType().equals("\n")){
            if(!tok.getType().equals("Integer")){
                System.out.println(tok + " is not integer");
                System.out.println("Invalid");
                return false;
            }
            tok = l1.getNextToken();
            System.out.println(tok);

            if (!tok.getType().equals("PlusOp")){
                System.out.println(tok + " is not identifier");
                System.out.println("Invalid");
                return false;
            }
            tok = l1.getNextToken();
        }

        return true;
    }

    public boolean parseAssignOp(){
        //l1.getNextToken();

        if(tok.getType().equals("AssignOp")){
            System.out.println("=");
            tok = l1.getNextToken();

            return true;
        }else{
            return false;
        }

    }

    public boolean parseId() {

        System.out.println(tok);
        l1.index++;
        //tok = l1.getNextToken();
        if (tok.getType().equals("Identifier")) {
            tok = l1.getNextToken();
            return true;
        }else {
            System.out.println("Not an Identifier");
        }

        return false;
    }

    public void parseAssignment(){
        //l1.getNextToken();
        parseId();
        parseAssignOp();
        parseExpression();

    }

    public void parseProgram(){

        while(tok.getType() != "End of File"){
            parseAssignment();
            break;

        }

    }

    public static void main(String [] args){

        Lexer l1 = new Lexer("/Users/swapnil/IdeaProjects/LexerParser/src/input.txt");
        Parser par = new Parser(l1);
        par.parseProgram();

    }
}

LexerParser [~/IdeaProjects/LexerParser]-,../src/Lexer.java [LexerParser] Project 『LoxerParser-/IdeaProjects/LexerParser idea npert jeva.lo. Ai leiotroundtxception ▼ import java.io.FileNotFoundException import java.io.IOException; out 수ǐnport java.io.FileReader; 4 션 input.txt Lexer Token LexerParser.im public class Lexer final int MAXSIZE 3800: char buffer[l; int bufferSize int index- ▶ llll External Libraries Scratches and Consoles public Lexer(String fildlane) { 15 this buffer -new char [MAXSIZE] getInput (fileName); 19 private void getInput(String fileNane) File inputFile new File(fileNane) try (FileReader reader new FileReader(inputFile) f this this.bufferSize- reader.read(this.buffer, offset: 0, length: 1800) } catch (F1 LeNotFoundException fnf) System.out.print ln( "File not found." System.out.print In(fnf.getMessage)) System.exit( status: 1) catch (IOException exc) Svstem.out.print In("TO Exception) Lexer getinputo Main, 뭍 Main /Library/Java/JavaVirtual Run: achines/jdk-9.0.4 dk/Contents/Home/bin/java avaagent/Applications/Intellij IDEA CE app/Contents/lib/idea 63877: /Applications/Intellil IDEA CE app/Contents/bi rt ar Type Token Value I Identifier 34 nteger 2928 LF: UTF-8 Compilation completed successfully in 2s 39ms (moments ago)