A Simple Mathematical Expression Tokenizer in C++

author@junian.dev (Junian Triajianto) — Tue, 16 Sep 2025 14:50:37 +0700

Back in college, I was given the task of implementing a math expression tokenizer function in C++.

It’s similar to LLM tokenization, but also quite different.

Given the following math expression as a string: (A+B)-9*X

The tokens are: [(, A, +, B, ), -, 9, *, X]

Another example with this math expression: 100+VARX+VARY-VARZ^2

The tokens are: [100, +, VARX, +, VARY, -, VARZ, ^, 2]

Here’s my implementation:

#include <iostream>

using namespace std;

bool isAlphabet(char chr)
{
    return ((chr >= 'a' && chr <= 'z') ||
            (chr >= 'A' && chr <= 'Z') ||
            chr == '$' || chr == '_');
}

bool isNumber(char chr)
{
    return (chr >= '0' && chr <= '9');
}

bool isOperator(char chr)
{
    return (chr == '+' || chr == '-' || chr == '/' || chr == '*' || chr == '^');
}

bool isBranch(char chr)
{
    return (chr == '(' || chr == ')');
}

bool isNull(char chr)
{
    return chr == '\0';
}

bool isSpace(char chr)
{
    return chr == ' ';
}

typedef struct MathTokenizer
{
    // the expression string
    string str;
    // pointer of char of expression
    char *pointer;

    // reset pointer to the first character of the expression
    void reset()
    {
        pointer = &str.at(0);
    }

    // get token from expression
    string getToken()
    {
        string result = "";

        // if pointer is null then end this process
        if (isNull(*pointer))
        {
            return "";
        }
        // if pointer is space then check next
        // ignore spaces
        else if (isSpace(*pointer))
        {
            pointer++;
            while (true)
            {
                if (isNull(*pointer))
                {
                    return "";
                }
                else if (isSpace(*pointer))
                {
                    pointer++;
                }
                else
                {
                    break;
                }
            }
        }

        // if operator or branch
        if (isOperator(*pointer) || isBranch(*pointer))
        {
            result = *pointer;
            pointer++;
        }
        // if variable / alphabet
        else if (isAlphabet(*pointer))
        {
            result = *pointer;
            pointer++;
            while (true)
            {
                if (isAlphabet(*pointer) || isNumber(*pointer))
                {
                    result += *pointer;
                }
                else
                {
                    break;
                }
                pointer++;
            }
        }
        // if number
        else if (isNumber(*pointer))
        {
            result = *pointer;
            pointer++;
            while (true)
            {
                if (isNumber(*pointer))
                {
                    result += *pointer;
                }
                else
                {
                    break;
                }
                pointer++;
            }
        }

        return result;
    }
} MathTokenizer;

int main()
{
    MathTokenizer exp;
    string s;

    while (true)
    {
        // input expression
        getline(cin, exp.str);
        exp.reset();

        // show tokens
        while ((s = exp.getToken()) != "")
        {
            printf("%s\n", s.c_str());
        }
    }

    return 0;
}

Get the complete source on my Git repository.

How To Make A 2D Animation using Timer with C++ .NET

author@junian.dev (Junian Triajianto) — Tue, 06 Dec 2016 21:09:00 +0000

On my last post (How To Draw 2D Primitive Objects on Windows Forms Application using C++ .NET), we’ve learned about how to draw some 2D objects on Form and Panel. That’s not difficult right? But now, in this post, we’ll learn about how to move that objects. Of course it’ll be very boring if that objects didn’t move. So, let’s get started! Don’t forget that I’m using Microsoft Visual Studio 2008 Professional Edition in this tutorial.

How To Draw 2D Primitive Objects with C++ .NET

author@junian.dev (Junian Triajianto) — Thu, 01 Dec 2016 20:47:00 +0000

First of all, I’d like to present my first tutorial about C++ / CLI CLR .NET Programming. This time it’s about Graphical User Interface in C++ .NET (in this case, I don’t want to use native C++). Okay, I assume that you have a little knowledge about C++ .NET GUI and of course, I also assume that you understand how to program in C++ CLR behaviour. So, here we go the tutorial!

C++ on Junian Dev

A Simple Mathematical Expression Tokenizer in C++

How To Make A 2D Animation using Timer with C++ .NET

How To Draw 2D Primitive Objects with C++ .NET