core.datasets.chat

core.datasets.chat

chat dataset module

Classes

Name	Description
TokenizedChatDataset	Tokenized chat dataset

TokenizedChatDataset

core.datasets.chat.TokenizedChatDataset(
    data,
    model_transform,
    *args,
    message_transform=None,
    formatter=None,
    process_count=None,
    keep_in_memory=False,
    **kwargs,
)

Tokenized chat dataset