{"id":2882,"date":"2026-07-09T11:14:34","date_gmt":"2026-07-09T11:14:34","guid":{"rendered":"https:\/\/justfineinfotech.com\/benchmarking-coding-agents-on-databricks-multi-million-line-codebase\/"},"modified":"2026-07-09T11:14:48","modified_gmt":"2026-07-09T11:14:48","slug":"benchmarking-coding-agents-on-databricks-multi-million-line-codebase","status":"publish","type":"post","link":"https:\/\/justfineinfotech.com\/fr\/benchmarking-coding-agents-on-databricks-multi-million-line-codebase\/","title":{"rendered":"Benchmarking Coding Agents on Databricks\u2019 Multi-Million Line Codebase"},"content":{"rendered":"

At Databricks, the way we build software is changing quickly as we aggressively adopt AI for engineering. The landscape of models and harnesses for code authoring has rapidly expanded in the last year, giving developers more choices than ever. With more options, it has become increasingly important to understand which coding agents offer the best performance on real-world coding tasks as well as understanding how task-performance varies with price.<\/p>\n

This article shares the results and methodology of the internal coding benchmark we built at Databricks, which evaluates tools on actual coding tasks our engineers performed on the Databricks codebase. Tasks featured edits against a multi-million line codebase covering many popular languages (Python, Go, Typescript, Scala, etc.) and both tasks and solutions were carefully reviewed to ensure accuracy. This isn’t meant to be comprehensive, but the exercise surfaced insights that have already made our engineering team meaningfully more efficient with coding agents. Below, you can see how models and harnesses scored on the overall benchmark:<\/p>\n

\"Cost <\/p>\n

Figure 1: Cost vs. Performance on our benchmark<\/strong><\/p>\n

The main conclusions from our analysis were:<\/p>\n

    \n
  1. The Pareto frontier for coding tasks (i.e. best quality for a given cost) includes models from OpenAI, Anthropic,\u00a0and<\/strong>open source. This means today, only a mix of tools can provide frontier performance.<\/li>\n
  2. Open models, and GLM 5.2 in particular, are now able to handle even the highest level of task difficulty.<\/li>\n
  3. The token price of a model is a poor indicator of actual costs incurred on end-to-end tasks. Larger models can be far more token efficient and have lower overall costs.<\/li>\n
  4. The harness a model is called from dramatically impacts cost and quality. In many cases, simple harnesses like Pi performed best on our workloads.<\/li>\n<\/ol>\n

    Let\u2019s dive a bit deeper on each one<\/p>\n

    Models cluster into rough \u201ccapability tiers\u201d<\/h3>\n

    Specific results being a couple points off can often even out in real world tasks. We focused more on the thematic patterns that help us reason about which models to use for various tasks. In fact, the results showed clear clustering of the models and harnesses into 3 capability tiers<\/p>\n

    \"Capabilities <\/p>\n

    Figure 2: Three distinct capability tiers emerged in our overall results, with nuance in which models were effective in each group<\/strong><\/p>\n

    At the upper end of performance, we see that the most intelligent models are very effective at solving all kinds of problems, but they\u2019re very expensive. Medium and lower intelligence models are still highly effective at the common tasks, and in many cases, they\u2019re also significantly cheaper.\u00a0<\/p>\n

    Day to day, engineers do a lot of different things that vary significantly in complexity: common operational tasks like flipping a flag or updating configs don\u2019t require extremely intelligent models, but deeper design explorations do. However, in the past, our default models were always the\u00a0most\u00a0<\/em>expensive ones. Based on this analysis we determined we should push more work to the Haiku and GPT 5.4 Mini class of models.<\/p>\n

    Open models are here for coding<\/h3>\n

    There\u2019s been a lot of excitement about GLM 5.2, and our results showed evidence that GLM can be a daily driver model for a lot of our developers. It landed in the top capability tier, statistically tied with Opus 4.8 on quality, but costing $1.28\/task against Opus\u2019s $1.94<\/p>\n

    The GLM quality scores are consistent with qualitative feedback we\u2019ve gotten from internal developers who have been piloting GLM for daily development. Because of its great performance for everyday coding tasks, we\u2019ve been focused on\u00a0serving GLM with the best performance<\/a>, and the evidence shows it\u2019s time to start deploying these as daily drivers for coding.\u00a0<\/p>\n

    Price-per-task vs price-per-token<\/h3>\n

    Developers often eyeball token costs to determine how expensive a model will be when completing coding tasks. We found, however, that token costs are often a poor indicator of overall task costs, due to variance in reasoning efficiency amongst models. This underscores the need for task-level benchmarking, since task shape and complexity may be different in different contexts<\/p>\n

    As an example, Sonnet 5 is ~1.7x cheaper per token than Opus 4.8, but, on our tasks, we found that Sonnet cost $2.09\/task vs Opus\u2019s $1.94, while scoring six points lower on task completion (81% vs 87%). This was mostly because Sonnet 5 worked longer and read more to get there, consuming 1.9x more tokens.\u00a0<\/p>\n

    Harnesses have a major impact on efficiency<\/h3>\n

    When we ran the same model with the same thinking effort through two different harnesses (Claude Code\/Codex vs Pi), we observed that the cost per task differed significantly (more than 2x in some cases), while quality remained the same. The main difference came down to how much context each harness fed the model on each turn<\/p>\n

    \"Harness <\/p>\n

    Pi sent about 3x less context per turn. It managed context better, keeping a tighter working set and finishing the tasks in fewer runs.\"Total<\/p>\n

    The lesson here isn\u2019t that one harness is always cheaper or that native harnesses are worse. Instead, model choice is only one piece of the puzzle. Establishing this flexibility is why we invested in\u00a0Omnigent<\/a>to make mode-and-harness swaps seamless.\u00a0<\/p>\n

    Why build your own benchmark?<\/h2>\n

    Public benchmarks like SWE-Bench and TerminalBench are useful, but they can\u2019t answer the questions we had. There are a few reasons for this:<\/p>\n