Use Amazon SageMaker HyperPod to fine-tune GPT-OSS-20B for multilingual reasoning.

GPT-OSS-20B, an open-weight reasoning model, defaults to thinking in English even when explicitly instructed to reason in French, German, Spanish, or Italian. On a 50-example multilingual reasoning test, the base model produced chain-of-thought in the correct target language only 22 percent of the time—and that 22 percent was almost entirely English examples (English 100 percent, French 0 percent, German 10 percent, Spanish 0 percent, Italian 0 percent). A team fine-tuning this model on Amazon SageMaker HyperPod set out to close that gap, comparing supervised fine-tuning (SFT) alone against SFT followed by reinforcement learning with Group Relative Policy Optimization (GRPO). Their results offer a practical framework for any team customizing open-weight reasoning models for non-English users.

Why the English Prior Matters

For teams deploying reasoning models to non-English users, a model that switches to English mid-thought creates a visible failure. A French-speaking user who asks a question in French expects the model to reason and answer in French. The base GPT-OSS-20B acknowledges the instruction but then produces its chain-of-thought in English anyway—a behavior rooted in its training distribution.

The evaluation used 50 test cases: 10 math and logic problems across five languages (English, French, German, Spanish, Italian). Two metrics mattered: whether the chain-of-thought was in the requested language, and whether the final answer was in the requested language. The base model scored 22 percent on reasoning-language correctness (11 out of 50), with near-total failure outside English.

SFT vs. GRPO: A Simple Mental Model

The distinction between the two techniques explains nearly every result in this experiment. SFT teaches a behavior the model does not have. GRPO optimizes a behavior the model can already sometimes produce. GRPO works by sampling a group of responses per prompt and pushing the policy toward above-average ones. If the desired behavior has zero probability in the base distribution, there is very little for GRPO to reinforce.

This framing guided the entire approach: first use SFT to establish multilingual reasoning where none existed, then apply GRPO only where SFT left measurable headroom.

The Training Stack

All fine-tuning ran on Amazon SageMaker HyperPod with Amazon Elastic Kubernetes Service (Amazon EKS) orchestration. The initial SFT and GRPO runs on GPT-OSS-20B used the PyTorch TRL library. A separate control experiment on Qwen2.5-7B switched to veRL as the trainer, with Ray and KubeRay for distributed orchestration and vLLM for fast rollout generation. Checkpoints and datasets lived on Amazon FSx for Lustre, and inter-node communication used Elastic Fabric Adapter (EFA). All fine-tuning used Low-Rank Adaptation (LoRA), which is parameter-efficient and compatible with both SFT and GRPO.

The data path worked as follows: a submission from the user landed on the Amazon EKS control plane of the SageMaker HyperPod cluster. Amazon EKS scheduled a Ray head pod plus worker pods running the veRL trainer and vLLM rollout workers. Both sets of pods mounted Amazon FSx for Lustre for the base model, training data, and checkpoints. Inter-node collectives used EFA. Rollouts streamed from vLLM back to the veRL trainer, which wrote checkpoints back to FSx for Lustre.

The GPT-OSS SFT run used 4× ml.g6e.12xlarge instances (16× L40S 48GB GPUs). The Qwen control used 2× g6e.48xlarge (16× L40S). The dataset was HuggingFaceH4/Multilingual-Thinking—a chain-of-thought reasoning dataset translated into French, Spanish, German, and Italian alongside English, with roughly 949 training and 50 validation examples.

SFT Results: Reasoning Language Fixed, Answer Language Lagging

The SFT configuration used LoRA r=8, α=16, trained for 1,000 steps with Fully Sharded Data Parallel (FSDP) full-shard, bf16 precision, and gradient checkpointing. The job ran as an elastic HyperPodPyTorchJob managed by Kueue.

After SFT, the model reasoned in the correct language 100 percent of the time (50 out of 50) across all five languages—a jump from 22 percent. The learning happened in two stages: around step 200, the model learned to reason in the target language; around step 1000, it additionally learned to give the answer in the target language.

But the final SFT evaluation revealed a split result. Reasoning-language correct: 100 percent (50/50). Answer-language correct: 62 percent (31/50). Both correct: 62 percent. Per-language answer accuracy: French 60 percent, German 60 percent, Spanish 30 percent, Italian 60 percent, English 100 percent.

The takeaway: SFT was not optional. The base model could not follow the reasoning-language instruction at all, and SFT took that metric from 22 percent to 100 percent. The remaining gap was answer-language at 62 percent—the harder half, because the final answer is short and the English prior is strongest there.

GRPO Adds 8 Points on Answer Language

Answer-language was the obvious target for GRPO. SFT already produced correct reasoning 100 percent of the time, and the model sometimes produced the answer in the right language, so there was a signal to reinforce.

The GRPO run used TRL's GRPOTrainer, starting from the SFT checkpoint. The setup loaded the SFT LoRA, called merge_and_unload, and added a new LoRA (r=8, α=16, about 0.068 percent trainable parameters—13.6M of 20B). Training used K=8 generations per prompt, device_map=auto, learning rate 1e-6, and 10 epochs. The reward function scored answer-language (±5.0), reasoning-language (±1.5), and brevity (+0.5 or −1.0), for a combined range of −7.5 to +7.0.

The result: reasoning-language correct held at 100 percent—GRPO did not break what SFT taught. Answer-language correct rose from 62 to 70 percent (+8 points, 35/50). Both correct: 70 percent. Per-language answer accuracy: French 70 percent, German 80 percent, Spanish 60 percent, Italian 40 percent, English 100 percent.

The gain came where there was headroom to gain. Answer-language was the sometimes-correct behavior that SFT had not fully converged on, and GRPO reinforced it.

What This Means for Teams Fine-Tuning Reasoning Models

The experiment points to a decision rule. Use SFT first to establish a behavior the model lacks entirely—in this case, reasoning in the target language. Then evaluate whether the remaining gap represents a behavior the model sometimes produces correctly. If yes, GRPO can add measurable value on top. If the behavior has near-zero probability in the base distribution, GRPO has little to reinforce.

The team also ran a controlled Qwen2.5-7B experiment across two tasks with different capability gaps to test this rule more broadly. The pattern held: SFT does most of the work, GRPO refines when there is headroom to refine, and matching the technique to the capability gap decides the outcome.

For teams deploying reasoning models to non-English users, the practical implication is that multilingual reasoning is a solvable fine-tuning problem—but the solution is staged. SFT alone took reasoning-language from 22 to 100 percent. GRPO added 8 points on answer-language. Neither technique alone would have delivered both results.

The full setup—including Kubernetes manifests, training scripts, and evaluation code—is available in the awsome-distributed-ai GitHub repository under the GPT-OSS LoRA GRPO test case. Teams can reproduce the pipeline on SageMaker HyperPod with EKS orchestration, FSx for Lustre storage, and EFA-enabled networking.