FormatPRM¶
帮助 Step 将数据转换为 PRM 模型期望的格式。
此 Step 可用于将数据格式化为 2 种格式之一： 遵循 peiyi9979/Math-Shepherd 中介绍的格式，在这种情况下，此 Step 创建列 input 和 label，其中 input 是带有解决方案的指令（以及被 token 替换的标签），label 是带有解决方案的指令，两者用换行符分隔。 遵循 TRL 用于训练的格式，该格式生成列 prompt、completions 和 labels。 labels 对应于被布尔值替换的原始标签，其中 True 表示正确的步骤。
属性¶
-
format: 用于 PRM 模型的格式。“math-shepherd” 对应于原始论文，而 “trl” 是为使用 TRL 训练模型而准备的格式。
-
step_token: 用作唯一 token 的字符串，表示预测步骤分数的位置。
-
tags: 表示正确和不正确步骤的标签列表。仅当它与
MathShepherdCompleter中的默认值不同时才需要告知。
输入和输出列¶
graph TD
subgraph Dataset
subgraph Columns
ICOL0[instruction]
ICOL1[solutions]
end
subgraph New columns
OCOL0[input]
OCOL1[label]
OCOL2[prompt]
OCOL3[completions]
OCOL4[labels]
end
end
subgraph FormatPRM
StepInput[Input Columns: instruction, solutions]
StepOutput[Output Columns: input, label, prompt, completions, labels]
end
ICOL0 --> StepInput
ICOL1 --> StepInput
StepOutput --> OCOL0
StepOutput --> OCOL1
StepOutput --> OCOL2
StepOutput --> OCOL3
StepOutput --> OCOL4
StepInput --> StepOutput
输入¶
-
instruction (
str): 任务或指令。 -
solutions (
list[str]): 包含任务解决方案的步骤列表。
输出¶
-
input (
str): 带有解决方案的指令，其中标签被 token 替换。 -
label (
str): 带有解决方案的指令。 -
prompt (
str): 带有解决方案的指令，其中标签被 token 替换。 -
completions (
List[str]): 以步骤列表形式表示的解决方案。 -
labels (
List[bool]): 标签，以布尔值列表形式表示，其中 True 表示良好的响应。
示例¶
Shepherd 格式¶
from distilabel.steps.tasks import FormatPRM
from distilabel.steps import ExpandColumns
expand_columns = ExpandColumns(columns=["solutions"])
expand_columns.load()
# Define our PRM formatter
formatter = FormatPRM()
formatter.load()
# Expand the solutions column as it comes from the MathShepherdCompleter
result = next(
expand_columns.process(
[
{
"instruction": "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
"solutions": [["Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +", "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can divide 2 by 2: 2 / 2 = <<2/2=1>>1 bolt of white fiber. +", "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"], ["Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +", "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can multiply 2 by 0.5 (which is the same as dividing by 2): 2 * 0.5 = <<2*0.5=1>>1 bolt of white fiber. +", "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"], ["Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +", "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can multiply 2 by 0.5 (which is the same as dividing by 2): 2 * 0.5 = <<2*0.5=1>>1 bolt of white fiber. +", "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"], ["Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +", "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can multiply 2 by 0.5 (which is the same as dividing by 2): 2 * 0.5 = <<2*0.5=1>>1 bolt of white fiber. +", "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"], ["Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +", "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can divide 2 by 2: 2 / 2 = <<2/2=1>>1 bolt of white fiber. +", "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"]]
},
]
)
)
result = next(formatter.process(result))
准备您的数据以使用 TRL 格式训练 PRM 模型¶
from distilabel.steps.tasks import FormatPRM
from distilabel.steps import ExpandColumns
expand_columns = ExpandColumns(columns=["solutions"])
expand_columns.load()
# Define our PRM formatter
formatter = FormatPRM(format="trl")
formatter.load()
# Expand the solutions column as it comes from the MathShepherdCompleter
result = next(
expand_columns.process(
[
{
"instruction": "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
"solutions": [["Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +", "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can divide 2 by 2: 2 / 2 = <<2/2=1>>1 bolt of white fiber. +", "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"], ["Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +", "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can multiply 2 by 0.5 (which is the same as dividing by 2): 2 * 0.5 = <<2*0.5=1>>1 bolt of white fiber. +", "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"], ["Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +", "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can multiply 2 by 0.5 (which is the same as dividing by 2): 2 * 0.5 = <<2*0.5=1>>1 bolt of white fiber. +", "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"], ["Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +", "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can multiply 2 by 0.5 (which is the same as dividing by 2): 2 * 0.5 = <<2*0.5=1>>1 bolt of white fiber. +", "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"], ["Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +", "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can divide 2 by 2: 2 / 2 = <<2/2=1>>1 bolt of white fiber. +", "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"]]
},
]
)
)
result = next(formatter.process(result))
# {
# "instruction": "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
# "solutions": [
# "Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required. +",
# "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can divide 2 by 2: 2 / 2 = <<2/2=1>>1 bolt of white fiber. +",
# "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3 +"
# ],
# "prompt": "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
# "completions": [
# "Step 1: Determine the amount of blue fiber needed: 2 bolts of blue fiber are required.",
# "Step 2: Calculate the amount of white fiber needed: Since it's half that much, we can divide 2 by 2: 2 / 2 = <<2/2=1>>1 bolt of white fiber.",
# "Step 3: Add the amount of blue and white fiber: 2 (blue) + 1 (white) = <<2+1=3>>3 bolts of fiber in total. The answer is: 3"
# ],
# "labels": [
# true,
# true,
# true
# ]
# }