what is the difference between a Lexeme and a Token? what do each represent?

Question

Question

Answer 1

Answer #1

Token: Tokens are the terminal symbols of the source language.

Eg: identifiers, numbers, keywords, punctuation etc.

Lexeme: They are the specific instance of the token. They are matched against the pattern (ex:- number [0-9])

for eg: count = count + temp ;

Here token is: Identifier, operator,punctuation

and lexeme are instances of token

identifier:-> count,temp

operator:-> =,+

punctuation:-> ;

Add a comment

Answer 2

Similar Homework Help Questions

Explain the difference between communication access techniques: - CSMA/CD - CSMA/CA - Token ring (5 marks)

Explain the difference between communication access techniques: - CSMA/CD - CSMA/CA - Token ring (5 marks)
Discuss the difference between and give examples that represent each type of Include a description of...

Discuss the difference between and give examples that represent each type of Include a description of rules for naming. This is worth 10 points, give me a 10 point covalent. answer.
what does the Fermi level of a material represent? Is there a difference between the terms...

what does the Fermi level of a material represent? Is there a difference between the terms "Fermi energy" and "Fermi level"? How does the Fermi level differ for metals, semiconductors, and insulators? What is the relationship between the Fermi level and the density of states

What is the difference between abilities and skills provide examples of each and how do these...

What is the difference between abilities and skills provide examples of each and how do these examples relate to job performance?
What is the difference between growth DISTANCE charts and growth VELOCITY charts? What do they each...

What is the difference between growth DISTANCE charts and growth VELOCITY charts? What do they each measure? What are some key differences between the charts (units on y-axis, at what ages is the highest point on the charts, or the lowest points, and what are their values, where are there increasing or decreasing slopes) and what do they tell us about growth?
What is the difference between an empirical question and a normative question? How do each type...

What is the difference between an empirical question and a normative question? How do each type of question apply to problems faced in American politics?

Class, What is the difference between a data entity (data model) and an external entity (process...

Class, What is the difference between a data entity (data model) and an external entity (process model)? Tables are made up of rows and columns. What do the rows represent? What do the columns represent? What is the difference between the logical ERD and the physical ERD? How can you translate entities from the CDFD? (Analyze the data flows.)
what's the difference between horizontal tank and vertical tank? what advantages do they have against each...

what's the difference between horizontal tank and vertical tank? what advantages do they have against each other?
Let X represent the difference between the number of heads and the number of tails obtained...

Let X represent the difference between the number of heads and the number of tails obtained when a coin is tossed n times. What are the possible values of X?

write a C++ program that reads in a text file and reads each character or lexemes...

write a C++ program that reads in a text file and reads each character or lexemes and states what token it is and writes to another file for example: while (fahr < upper) a = 23.00 whileend Output: token lexeme -------------------------------- keyword while separator ( identifier fahr operator < identifier upper separator ) identifier a operator = real 23.00 keyword whileend the program aslo reads in comments but does not add to the list. characters that are commented out or...