prompt_tokenizers

prompt_tokenizers

Module containing PromptTokenizingStrategy and Prompter classes

Classes

Name	Description
AlpacaMultipleChoicePromptTokenizingStrategy	Tokenizing strategy for Alpaca Multiple Choice prompts.
AlpacaPromptTokenizingStrategy	Tokenizing strategy for Alpaca prompts.
AlpacaReflectionPTStrategy	Tokenizing strategy for Alpaca Reflection prompts.
DatasetWrappingStrategy	Abstract class for wrapping datasets for Chat Messages
GPTeacherPromptTokenizingStrategy	Tokenizing strategy for GPTeacher prompts.
InstructionPromptTokenizingStrategy	Tokenizing strategy for instruction-based prompts.
InvalidDataException	Exception raised when the data is invalid
JeopardyPromptTokenizingStrategy	Tokenizing strategy for Jeopardy prompts.
NomicGPT4AllPromptTokenizingStrategy	Tokenizing strategy for NomicGPT4All prompts.
OpenAssistantPromptTokenizingStrategy	Tokenizing strategy for OpenAssistant prompts.
PromptTokenizingStrategy	Abstract class for tokenizing strategies
ReflectionPromptTokenizingStrategy	Tokenizing strategy for Reflection prompts.
SummarizeTLDRPromptTokenizingStrategy	Tokenizing strategy for SummarizeTLDR prompts.

prompt_tokenizers.AlpacaMultipleChoicePromptTokenizingStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Tokenizing strategy for Alpaca Multiple Choice prompts.

prompt_tokenizers.AlpacaPromptTokenizingStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Tokenizing strategy for Alpaca prompts.

prompt_tokenizers.AlpacaReflectionPTStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Tokenizing strategy for Alpaca Reflection prompts.

prompt_tokenizers.DatasetWrappingStrategy()

Abstract class for wrapping datasets for Chat Messages

prompt_tokenizers.GPTeacherPromptTokenizingStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Tokenizing strategy for GPTeacher prompts.

prompt_tokenizers.InstructionPromptTokenizingStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Tokenizing strategy for instruction-based prompts.

prompt_tokenizers.InvalidDataException()

Exception raised when the data is invalid

prompt_tokenizers.JeopardyPromptTokenizingStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Tokenizing strategy for Jeopardy prompts.

prompt_tokenizers.NomicGPT4AllPromptTokenizingStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Tokenizing strategy for NomicGPT4All prompts.

prompt_tokenizers.OpenAssistantPromptTokenizingStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Tokenizing strategy for OpenAssistant prompts.

prompt_tokenizers.PromptTokenizingStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Abstract class for tokenizing strategies

prompt_tokenizers.ReflectionPromptTokenizingStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Tokenizing strategy for Reflection prompts.

prompt_tokenizers.SummarizeTLDRPromptTokenizingStrategy(
    prompter,
    tokenizer,
    train_on_inputs=False,
    sequence_len=2048,
)

Tokenizing strategy for SummarizeTLDR prompts.

Name	Description
parse_tokenized_to_result	Parses the tokenized prompt and append the tokenized input_ids, attention_mask and labels to the result
tokenize_prompt_default	Returns the default values for the tokenize prompt function

prompt_tokenizers.parse_tokenized_to_result(
    result,
    current_len,
    res,
    labels,
    pad_token_id=None,
)

Parses the tokenized prompt and append the tokenized input_ids, attention_mask and labels to the result

prompt_tokenizers.tokenize_prompt_default()

Returns the default values for the tokenize prompt function