A direct prompt came within three points of a multi-step agent in a five-way test of prompt engineering techniques on the same pull request task. The video by Gustavo Dev Doido runs each style against identical inputs, then scores every output with an LLM judge.
Five prompt engineering techniques, one pull request task
A September 2026 video from the Full Cycle channel, presented by Gustavo Dev Doido, tested five prompt engineering techniques on the same task: writing a pull request from a code diff, an issue, a pull request template, repository conventions, and passing CI test results. Each technique ran with LangChain against an OpenAI reasoning model, and the outputs were ranked by an LLM judge.
The setup matters because it holds everything constant except the prompt. Same inputs, same model, same goal. That makes it a clean, if informal, way to see what prompt structure adds and what it costs. The experiment is a teaching exercise from the Full Cycle ecosystem, which includes the Crazystack Typescript platform at crazystack.com.br and the Bootcamp do Dev Doido, not a published benchmark.
The five techniques were: direct prompt, chain of thought, skeleton of thought, tree of thoughts, and a ReAct agent with tools. The shared system prompt instructed the model to act as a senior developer, use only evidence-backed facts, and return the result in Markdown.
What is a direct prompt?
A direct prompt is a single instruction with no reasoning scaffold: you describe the role, hand over the inputs, and ask for the output. In the experiment, the system prompt said the model was a senior developer creating a precise pull request from the supplied issue, diff, test results, and repository conventions.
This baseline produced a usable pull request in one model call. The generated document pulled in the diff, referenced the issue, and described the tests. If you only ever use one technique, this is probably the one, and the later scores show why it is hard to beat on cost.
How does chain of thought prompting work here?
Chain of thought prompting, introduced by Wei et al. in the 2022 paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, asks the model to reason step by step before answering. The presenter notes that a reasoning model already does some of this internally, so his version makes the steps explicit and auditable.
The prompt prescribed an ordered sequence: identify the business intent and expected behavior, map the issue's requirements onto concrete diff changes, verify that the tests actually prove something, identify recurring risks, then draft the pull request from the repository template. The model worked through exactly those stages, surfacing details like idempotency for duplicate-payment prevention, and exposed its notes in reviewer-friendly form.
The benefit the presenter highlighted is auditability. Because each intermediate conclusion is written down, you can check whether the reasoning behind a pull request line is sound, instead of trusting an opaque one-shot answer.
What is skeleton of thought prompting?
Skeleton of thought prompting inverts the writing order. The model first produces a skeleton: a title plus a list of sections, where each section has a name, an objective, and the concrete evidence that supports it. Only then does a second loop expand each section, one model call per section, with instructions to be concise, technical, and evidence-bound.
The technique is described in the 2023 paper Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation, where it was proposed partly because section-level generation can run in parallel. In this experiment the payoff showed up as completeness: the presenter observed that each expanded section, such as how-to-test or rollout risks, read more fully developed than in the single-shot outputs.
The cost is call count. Where the direct prompt makes one call, the skeleton approach makes one call plus one per section, which means more latency and more tokens for the same pull request.
How does tree of thoughts improve the output?
Tree of thoughts, from Yao et al.'s 2023 paper Tree of Thoughts: Deliberate Problem Solving with Large Language Models, generates multiple candidate solutions, scores them against explicit criteria, expands the survivors, and repeats until a final version emerges.
The implementation asked the model for three different approaches to the same pull request: behavior-first, architecture-and-risk-first, and reviewer-first. Each candidate was scored on correctness, evidence grounding, usefulness for reviewers, and clarity. In the recorded run, the three candidates tied, two were kept, each produced a variation, and the winning variation became the final pull request.
The presenter's point is that you must define what good and bad mean. Without an explicit scoring rubric, a model has no principled way to pick between its own candidates. With one, it performs an internal vote that you can inspect.
What does a ReAct agent add?
A ReAct agent, based on the 2022 paper ReAct: Synergizing Reasoning and Acting in Language Models, differs from every technique above in one way: instead of receiving all inputs in the context, it gets tools and decides when to use them.
The agent received five tools: read the issue, read the diff, read the test results, read the pull request conventions, and save the pull request. Its instructions were to inspect every piece of evidence before writing anything, never invent implementations, and call the save tool exactly once. The traced run showed the agent reading all four inputs, observing their contents, saving the pull request, and reporting where it went.
This is the pattern that scales beyond a fixed input set. If your evidence lives in files, tickets, or CI systems, tools let the model pull what it needs rather than you pasting it. For a single well-defined run, though, the video's ranking shows that advantage did not convert into a better document.
Which technique won the ranking?
The presenter built a sixth script that fed all five generated pull requests to an LLM judge, which scored each on grounding, gap analysis, risk coverage, reviewer usefulness, clarity, and template adherence, six points available per criterion. The result, reported from the video's own experiment, was a near tie.
This is speaker-reported data from a single run with one model, so treat it as a demonstration rather than a benchmark. The gap between the cheapest and most elaborate techniques was three points out of thirty.
| Technique | Judge score (out of 30) | Model calls |
|---|---|---|
| Skeleton of thought | 30 | One per section plus skeleton |
| Tree of thoughts | 30 | Multiple, for candidates and scoring |
| Direct prompt | 27 | One |
| Chain of thought | 27 | One |
| ReAct agent | 27 | One per tool use |
The presenter's own conclusion is the useful one: a sophisticated prompt is not automatically the best prompt for a given criterion. Match the technique to the job. Use direct prompting when a single call suffices, chain of thought when you need auditable reasoning, skeleton of thought for fuller sections, tree of thoughts when candidate diversity matters, and an agent when the evidence must be gathered rather than supplied.
FAQ
- Do reasoning models still need chain of thought prompts? Partially. The presenter notes that a reasoning model already chains internally, but an explicit chain of thought prompt makes each step visible so you can audit how the model moved from diff to conclusion.
- Was this a scientific benchmark? No. It was one presenter's experiment, run once in September 2026 with one model on one pull request task, and judged by another LLM. The scores illustrate relative behavior, not measured performance across tasks.
- Which technique should I start with? The direct prompt. It scored 27 of 30 in this experiment with a single model call, and you should only add structure when a specific weakness, such as thin sections or unverifiable reasoning, justifies the extra calls.
- Why give an agent tools instead of the full context? Tools let the model fetch exactly the evidence it needs from files or systems, which scales better than pasting everything into the prompt. In this single fixed-input run, that flexibility did not produce a higher score.
Turn your own walkthroughs into written guides
The real lesson from this experiment is that a clear, structured explanation of five techniques is worth more than the scores themselves, and that kind of explanation often lives in a twenty-minute video where only part of the audience takes notes. If you record walkthroughs like this, Skala Blog turns that existing material into a written article: paste the YouTube video URL into Skala Blog, the video gets transcribed, and the transcript becomes an editable, structured article your readers can search and cite.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits