MathShepherdCompleter¶
Math Shepherd Completer 和自动标注器任务。
此任务负责，给定指令的一系列解决方案和一个作为参考的黄金解决方案，为这些解决方案生成补全，并使用参考论文图 2 中的硬估计方法（公式 3）根据黄金解决方案对其进行标记。这些属性使任务能够灵活地用于不同类型的数据集和 LLM，并允许使用不同的字段来修改系统和用户提示。在修改它们之前，请检查当前默认值以确保正确生成补全。
属性¶
-
system_prompt：在补全中使用的系统提示。默认提示已检查，并使用 Llama 3.1 的 8B 和 70B 模型生成良好的补全，但可以对其进行修改以适应所选的模型和数据集。
-
extra_rules：此字段可用于插入与数据集类型相关的额外规则。例如，在原始论文中，他们使用了 GSM8K 和 MATH 数据集，此字段可用于插入 GSM8K 数据集的规则。
-
few_shots：少量示例，以帮助模型生成补全，请以您数据集所需解决方案类型的格式编写它们。
-
N：每个步骤要生成的补全数量，对应于论文中的 N。他们在论文中使用了 8，但可以调整。
-
tags：要在补全中使用的标签列表，默认标签为 ["+", "-"]，如论文中所述，其中第一个用作正标签，第二个用作负标签。这可以更新，但它必须是一个包含 2 个元素的列表，其中第一个是正标签，第二个是负标签。
输入和输出列¶
graph TD
subgraph Dataset
subgraph Columns
ICOL0[instruction]
ICOL1[solutions]
ICOL2[golden_solution]
end
subgraph New columns
OCOL0[solutions]
OCOL1[model_name]
end
end
subgraph MathShepherdCompleter
StepInput[Input Columns: instruction, solutions, golden_solution]
StepOutput[Output Columns: solutions, model_name]
end
ICOL0 --> StepInput
ICOL1 --> StepInput
ICOL2 --> StepInput
StepOutput --> OCOL0
StepOutput --> OCOL1
StepInput --> StepOutput
输入¶
-
instruction (
str)：任务或指令。 -
solutions (
List[str])：任务的解决方案列表。 -
golden_solution (
str)：任务的参考解决方案，将用于注释候选解决方案。
输出¶
-
solutions (
List[str])：与用作输入的列相同，“solutions”已修改。 -
model_name (
str)：用于生成修订的模型名称。
示例¶
使用结构化输出通过 Math Shepherd Completer 注释您的步骤（首选方式）¶
from distilabel.steps.tasks import MathShepherdCompleter
from distilabel.models import InferenceEndpointsLLM
llm=InferenceEndpointsLLM(
model_id="meta-llama/Meta-Llama-3.1-8B-Instruct",
tokenizer_id="meta-llama/Meta-Llama-3.1-8B-Instruct",
generation_kwargs={
"temperature": 0.6,
"max_new_tokens": 1024,
},
)
task = MathShepherdCompleter(
llm=llm,
N=3,
use_default_structured_output=True
)
task.load()
result = next(
task.process(
[
{
"instruction": "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
"golden_solution": ["Step 1: Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.", "Step 2: She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s market.", "The answer is: 18"],
"solutions": [
["Step 1: Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.", "Step 2: She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s market.", "The answer is: 18"],
['Step 1: Janets ducks lay 16 eggs per day, and she uses 3 + 4 = <<3+4=7>>7 for eating and baking.', 'Step 2: So she sells 16 - 7 = <<16-7=9>>9 duck eggs every day.', 'Step 3: Those 9 eggs are worth 9 * $2 = $<<9*2=18>>18.', 'The answer is: 18'],
]
},
]
)
)
# [[{'instruction': "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
# 'golden_solution': ["Step 1: Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.", "Step 2: She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer\u2019s market.", "The answer is: 18"],
# 'solutions': [["Step 1: Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day. -", "Step 2: She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer\u2019s market.", "The answer is: 18"], ["Step 1: Janets ducks lay 16 eggs per day, and she uses 3 + 4 = <<3+4=7>>7 for eating and baking. +", "Step 2: So she sells 16 - 7 = <<16-7=9>>9 duck eggs every day. +", "Step 3: Those 9 eggs are worth 9 * $2 = $<<9*2=18>>18.", "The answer is: 18"]]}]]
通过 Math Shepherd Completer 注释您的步骤¶
from distilabel.steps.tasks import MathShepherdCompleter
from distilabel.models import InferenceEndpointsLLM
llm=InferenceEndpointsLLM(
model_id="meta-llama/Meta-Llama-3.1-8B-Instruct",
tokenizer_id="meta-llama/Meta-Llama-3.1-8B-Instruct",
generation_kwargs={
"temperature": 0.6,
"max_new_tokens": 1024,
},
)
task = MathShepherdCompleter(
llm=llm,
N=3
)
task.load()
result = next(
task.process(
[
{
"instruction": "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
"golden_solution": ["Step 1: Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.", "Step 2: She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s market.", "The answer is: 18"],
"solutions": [
["Step 1: Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.", "Step 2: She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s market.", "The answer is: 18"],
['Step 1: Janets ducks lay 16 eggs per day, and she uses 3 + 4 = <<3+4=7>>7 for eating and baking.', 'Step 2: So she sells 16 - 7 = <<16-7=9>>9 duck eggs every day.', 'Step 3: Those 9 eggs are worth 9 * $2 = $<<9*2=18>>18.', 'The answer is: 18'],
]
},
]
)
)
# [[{'instruction': "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
# 'golden_solution': ["Step 1: Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.", "Step 2: She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer\u2019s market.", "The answer is: 18"],
# 'solutions': [["Step 1: Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day. -", "Step 2: She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer\u2019s market.", "The answer is: 18"], ["Step 1: Janets ducks lay 16 eggs per day, and she uses 3 + 4 = <<3+4=7>>7 for eating and baking. +", "Step 2: So she sells 16 - 7 = <<16-7=9>>9 duck eggs every day. +", "Step 3: Those 9 eggs are worth 9 * $2 = $<<9*2=18>>18.", "The answer is: 18"]]}]]