utils.tokenization

utils.tokenization

Module for tokenization utilities

Functions

Name	Description
color_token_for_rl_debug	Helper function to color tokens based on their type.
process_tokens_for_rl_debug	Helper function to process and color tokens.

utils.tokenization.color_token_for_rl_debug(
    decoded_token,
    encoded_token,
    color,
    text_only,
)

Helper function to color tokens based on their type.

utils.tokenization.process_tokens_for_rl_debug(
    tokens,
    color,
    tokenizer,
    text_only,
)

Helper function to process and color tokens.