Price Per Token Is the Wrong Unit Now

You have probably seen that bar chart at least once, the one everybody quotes whenever a new AI model ships. Plenty of people know the chart without ever learning the name of the company behind it. The team that makes it, Artificial Analysis, came to Korea from San Francisco. They sit inside no lab, and every result they produce is published openly on their website.

This time we also invited two teams that build models here in Korea. SK Telecom and Trillion Labs each walked through how their own models come together, and George Cameron, co-founder of Artificial Analysis, joined by video at half past three in the morning San Francisco time to give the keynote. Three sessions in all, one from the people who measure the models and two from the people who make them.

Eighty-eight percent said they already read the benchmarks

Two hundred and seventy one people signed up, and the fifth floor of dcamp Mapo filled right up. The demographics we pulled out of the registration form explained the topic on their own. Eighty-eight percent said they already look at benchmarks when they choose a model, and thirty-six percent train models themselves or wire them into a product.

We also put the guest list on screen so people could see who else was sitting in the room, and reading it out loud took a while. From the model and AI side, SK Telecom and Trillion Labs were joined by NAVER and NAVER Cloud, Upstage, KT, KRAFTON AI, NC AI, Lablup, FriendliAI, Moreh, Maum AI, TwelveLabs, RLWRLD, Coxwave, VESSL AI, VUNO, Cochl and Mind AI. From the enterprise side the seats went to Samsung Electronics, Hyundai Motor, Hyundai Mobis, Hyundai AutoEver, LG CNS, LG Electronics, HD Hyundai Robotics, Coupang, Toss, Lotte Innovate, CJ ENM, SK Innovation, SK On, TMAP Mobility, Samsung Medical Center and Pulmuone.

The global companies brought Google Korea, Microsoft, AWS, Akamai, Arize AI, Sendbird, Photoroom, Mysten Labs, eBay Japan, Concentrix, Megazone Cloud, Cloocus and Bespin Global. From academia we had KAIST, Seoul National University, Yonsei, Korea University, Hanyang, Sungkyunkwan, Seoul National University of Science and Technology and Oxford. From media and investment came Chosun Ilbo, JoongAng Ilbo, Electronic Times, AI Times and VentureSquare, along with Samsung Venture Investment and Mirae Asset Capital. Very few of those names would normally turn up on the same registration list.

We left a box on the form for questions and 227 people wrote one in. How a score is actually built drew 55 of them, the distance between benchmark scores and real performance drew 45, Korean models and sovereign AI evaluation drew 29, choosing a model under cost and speed constraints drew 22, agent evaluation drew 14, and contamination and overfitting drew 9. Three of them went on screen word for word. When an index updates a model can score higher without changing at all, so how do you separate that from a real improvement? How should the performance of sovereign AI be measured? And how much does a benchmark score really shape a product decision?

The magic still comes from the model rather than the harness

Henry Jin, who leads Asia Pacific partnerships, opened by introducing the team. Artificial Analysis is an independent AI benchmarking and analysis company that evaluates the whole stack, from the model through inference down to the hardware layer, and the aim is to hold intelligence, speed and price in view together so developers and enterprises can reach the best informed decision available to them. Over the past year the team had been meeting Korean AI labs, government bodies and the developer and research community, and he said they had watched for themselves how quickly Korean AI moved. That was a large part of why they wanted to hold this event.

George followed with a short introduction to the company. The part he was proudest of was that companies quote the intelligence index bar chart to show the trade-offs a developer has to accept, rather than quoting a single ranking on its own, which is why all of those charts are 2x2. OpenAI took intelligence and cost, Jensen Huang took intelligence and speed, and Anthropic took a coding agent benchmark against a speed benchmark. Three different companies, three different pairs of axes, all pulled from the same source.

His opening point landed well. We mostly use these models as agents now, he said, though it is worth being clear about where the magic actually comes from. A harness hands the model tools that touch the real world, like code execution or web search, and what is possible is still pulled along by the model itself. For evidence he showed the coding agent index, which puts models into harnesses and tests both together. A good harness on a weak model did not beat an ordinary harness on a good model. A few years ago that was an open question, and so far it has held.

Running the top model on every task would break the company

The intelligence index combines ten evaluations that run from general knowledge and hallucination through scientific reasoning, coding and agentic workloads, and the leaders are the latest models from Anthropic and OpenAI. He put the ranking up and did not dwell on it. He said that would surprise nobody in this room, then added that what he wanted to talk about was why reasons to use other models still remain.

For an enterprise, using the top model for every task could bankrupt the company.

Choosing a model right now, in other words, is a complicated trade-off between competing considerations.

Korea now sits behind the United States and China

The Korea section drew the most nodding heads of the night. Over the past year Korea has climbed into the top three behind the United States and China. Countries like Canada or Spain have one or two individual models, he said, while Korea has several AI companies with models of their own, and it happened quite fast. He gave the credit to the companies that built them and to the Korean government's sovereign AI initiative together, and said Artificial Analysis had joined that effort, signed an MOU with NIPA, and planned to keep supporting it as an independent evaluation partner. That was the line people quoted back to me afterwards.

He put the progress of open weight models in a different frame. Back in early 2024, when the conversation was about Llama 2 70B or Mixtral 8x7B, whether open weights would stay competitive and whether labs would keep swallowing the cost of releasing them were both open questions. What has been settled since is that labs keep finding reasons to release, because open weights answer a real demand for flexibility from users who want to run or fine-tune a model on their own hardware. They will not overtake proprietary models any time soon, though they will keep delivering intelligence just below the frontier.

Price per token has become an outdated unit

When he reached the point that 2026 is about more than intelligence, he slipped in a joke. The Artificial Analysis site covered intelligence, speed and cost from the first day, and only this year did people start caring about the second and third charts. The site had been waiting for the rest of us to catch up.

Cost matters now because the spread between the leading models and the cheap ones runs to a hundred times. On the screen from 14 September that we showed in the opening, the cost of solving the same single task ranged from six cents up to eight dollars and seventy five cents. Finishing the tasks in the intelligence index costs more than seven dollars with the top model, while other models that still do the job very well come in under ten cents.

One of the inputs behind that cost is the number of output tokens a model spends to finish a single task, which turns on how verbose the model is and how many agentic steps it takes. Some models take twenty steps on a task and others take a hundred, and moving around a complicated environment, checking this file, checking whether it is following this policy, chews through an enormous number of tokens. None of that step count appears anywhere on a price page. The AI labs may not be eager to have you think this way, he added, since they charge by the token.

You cannot just use price per token any more. That is outdated. In a world of agents you have to think about the volume of output tokens and the cost per task.

Cost per task, the way he defined it, takes in answer tokens and reasoning tokens, cache writes and cache hits, and the input tokens as well.

Forces pushing cost down and forces pushing it up run at once

Two sets of forces run through the ecosystem at the same time, one pushing cost down and one pushing it up, and the split between them produces the hundredfold spread we see today. On the downward side sit smaller models and sparsity, which cut active parameters and compute per token, software efficiency gains that cut compute further, and hardware, where at scale a Blackwell chip is cheaper than Hopper while it handles more tokens. On the upward side sit bigger models, since demand for higher intelligence goes unsatisfied and everyone gets excited about each new release, and on top of that come more reasoning tokens and agents that make more requests per task.

One pattern the efficiency side produces was the practical part for me. Once a level of intelligence has been reached, that intelligence gets dramatically cheaper very quickly. They tracked each new intelligence index threshold, the price at the moment it was reached and what happened afterwards. Intelligence index 10 was reached in the middle of 2024, and the cost fell more than a hundred times after that. The same pattern repeats at other levels, where new intelligence arrives, competition follows and efficiency drags the price down behind it.

If the top model feels too expensive for what you want to do with it right now, you may not have to wait long for it to become usable.

That matters when you plan around what next year's costs will make possible, he said, and added that it is a decent lens for anyone building a startup.

Drawing the curve for your own workload instead of a general index

Mapping the intelligence and cost trade-off at a high level leaves a wide Pareto frontier between the leading models at over seven dollars a task and the much cheaper ones. You should be choosing along that frontier, and reasons to pick a different model still exist, like a workload that values intelligence differently, or a requirement for open weights, or a requirement for a model from a particular country. Each of those pulls you off the frontier for a reason you can defend.

That led into a new product. An intelligence benchmark is a composite, general purpose measure, so it works well for high level understanding, while everybody has a workload of their own. They recently released Optima, which lets anyone build a custom benchmark and receive a Pareto curve for their own workload. You log in on the site, upload the context of your use case, and it generates a benchmark and hands back a trade-off chart across models. He said it can save more than ten times the cost and the time, and that the expertise they built running frontier evaluations goes into it.

He also sketched what comes next. The way coding agents rewrote a developer's entire workflow from end to end is now spreading into general work across the rest of the economy. Continuous learning has already started in the shape of rules files, though it has not reached the point where a model updates its own weights. Models are turning natively multimodal while the way we use them shifts at the same time, and the larger move is away from reacting and toward acting far more on their own. We expect a person to come back, ask questions and gather context, so expecting the same of a model is fair.

Nobody measures Korean fluency properly yet

The first question of the Q&A was the sharpest of the night. There are a great many languages besides English, someone said, so how do you evaluate performance in them, and does human subjectivity not slip in where objective performance evaluation would not? He offered the rest as a personal observation. Ask a model to express something in Korean and English idioms and English proverbs come out, which works badly for Korean sensibility and for what a Korean reader grasps intuitively, and without deliberate control the output drifts there on its own.

George answered in two layers. Basic understanding of a language is something these models abstract very well, because they abstract the concept of language itself in latent space, so they handle different languages natively and he sees little difference between languages in the ability to actually complete a task. That part of the answer was confident. Then came the honest half, which was that the nuance in how these models communicate is something current benchmarks fail to capture, and that this applies to the ones his own team built as much as to anyone else's.

I do not think this topic is being handled well. Honestly, that includes us.

He said it is an important area, that he would like to see more research and would like his team to publish more of it, and he asked the community to help build benchmarks in that space. A personality benchmark is possible, and it would probably start in English, so he said it would be great if the international community looked at it and added other language perspectives.

More of the metrics ahead will stop being higher is better

The next question came from someone who uses AI benchmarks heavily at work. You said agents will get far more capable, he began, so from the position of someone who runs evaluations, where is human-made data or human judgment still needed the most today, and what falls away first?

The past year to eighteen months has been concentrated on reinforcement learning and on finishing tasks, George answered, and he split the data question in two. Getting models to take on new tasks will keep taking the shape of more complicated reinforcement learning environments, and there is a long way left to go. Right now we expect a model to successfully finish a very clearly defined task, and it does not go out on its own, gather context actively, ask the right questions and discover something new. We are working inside a narrow band.

Making models better at what they already do needs more human data. It goes into improving how a model communicates and how it works alongside us, with reinforcement learning environments attached to that. From there he expects we will talk far more than we do today about metrics where higher is not automatically better, like efficiency, cost efficiency, time efficiency and communication style. One person prefers this communication style and another person prefers that one. The intelligence index will stay a higher-is-better measure, while finer grained metrics that help us understand models will keep growing up around it.

The last question was about the roadmap at the edge. Someone who benchmarks edge devices asked about robotics and wearables and whether there were plans to expand on mobile. He wants to broaden it, he said, though the AI world is large and still growing, so it turns into a question of priorities about what they can realistically build. Edge benchmarking will stay roughly where it is for now, while physical AI and robotics benchmarks will multiply as the surface area of devices running AI widens. He had spent the first two days of that week in Korea and had to get back to San Francisco for something urgent, and said he would gladly come in person next time.

After a usable model came a model you can really use

The second session belonged to Sungjun Cheon of SK Telecom. The subject was the thinking behind A.X K2, released this July, in the second round of the national foundation model program. SK Telecom was picked as one of the elite teams for the independent AI foundation model program run by the Ministry of Science and ICT, and it is one of the three teams that passed the second stage of evaluation. The talk was titled Toward a Production-Ready 688B-A33B MoE, and the weight of it sat on the words Production-Ready rather than on the parameter count.

This was not his first time on that stage. In April, at the NVIDIA Nemotron developer day in the same venue, he had said the first round produced models that were usable and that the second round was about building a model you can really use. He came back this time with that promise in hand, which is a harder way to give a talk than most people choose.

He explained what it takes to reach Production-Ready by tying it to the question that had come earlier. There is nuance a benchmark cannot evaluate, he said, and only a person looking at it directly can judge it, since AI models still struggle to assess that properly. A Production-Ready model comes out of development that keeps catching nuance and usability along with everything else. He then picked up George's point about efficiency mattering more and more, and said the plan is to build models that grow steadily more efficient from the architecture through the training process and on into deployment.

They could not shrink the model, so the architecture had to work

A.X K2 has 688B parameters with 33B active, and it went out under Apache 2.0. The feedback they heard most after releasing the first round model was that it was too big to use, which is really a worry about cost, so it had to get faster and had to run inference on fewer GPUs. At the same time the scale of foundation model competition keeps climbing, with trillion-parameter models opening up outside and rumors of more than 10T inside the frontier labs. Shrinking the model was off the table, so the answer had to come out of architecture design instead.

The diagram he drew himself held every distinctive piece. The Indexer and Selector make attention sparse. A MoE is a sparse model by default, and adding sparsity to attention itself holds the cost of reading context near the number of selected tokens instead of a square. MLA uses a compressed latent space to shrink the KV cache. GatedNorm and the Attention Output Gate cut outliers in parameter space and stop attention sinks from being abused. Choose fewer tokens, store less state, control the outliers and the sinks. He said they do not claim every component as their own, while combining them this way is a novel proposal.

Building a new structure made the road to fast training and deployment hard going, he said. Speeding up training meant reaching down to the kernel level, and when the kernel paths for training and inference did not run the way they wanted, they squeezed out efficiency with complicated techniques like kernel fusion. None of that work shows up in a benchmark table.

What they observed became the spec for the next training run

For usability, the thing they paid special attention to this time was getting as much improvement as possible inside a single training loop. Compute was not plentiful enough to run it several times, so the one run they had needed to go well. The loop turned through five stages. They watched model responses in internal usability tests and in Arena, they wrote up capability gaps and response patterns as failure analysis, they moved data, rewards and safety goals into training design, they updated with RLHF and RLVR, and they re-evaluated with benchmarks, users and red teams before circling back to failure analysis.

Observed behavior became the specification for the next training iteration.

They placed people well inside that loop, with humans actually running the usability evaluation and pulling the improvements out of it. They built internal benchmarks and custom benchmarks, and they red-teamed both in house and through outside sourcing. He said they have kept growing the ability to improve a model continuously and the ability to do it with their own hands, and that watching usability improve meaningfully this time was the result that meant the most to them, more than any single score.

AIME at 97.1 and long context up by forty points

When he got to results he said people ask a lot about scores, and then he gave the numbers. At release two months ago the model was tied for first on AIME 2026 by MathArena, and he checked once more before coming out that day and found it still tied for first at 97.1. Internal testing also confirmed a level matching a gold medal at the 2026 International Mathematical Olympiad, though that one was not in the deck. Long context had been weak in the first round, and after hard work on it they reached 66.0 on AA-LCR, forty points above round one. In Korean they confirmed the lead again, with 80.5 on KMMLU-Pro and 91.6 on CLIcK.

How can a model that grew from 571B in the first round to 688B be more efficient and faster at the same time? The question comes up naturally, and the answer was that they trained in FP8. Up to the first round they trained in BF16, while this time training ran natively in eight bits, and they quantized further and released FP4 alongside it. GatedNorm and the sparse gated attention hold down the outliers that inflate quantization error, which is what made that choice possible in the first place.

Minimum model memory shows exactly what the design bought them. The first round in BF16 needed 1038GB, while A.X K2 needs 646GB in FP8 and 370GB in NVFP4, and NVFP4 reaches quality comparable to FP8 while serving on four NVIDIA B200 GPUs. They support GGUF too, which widens runtime compatibility and local deployment options. On the inference speed side they opened up both kinds of speculative decoding. EAGLE3 raises throughput by up to 1.64 times and cut time per output token by 28 percent on production traffic, and DSpark runs about 1.8 times faster and cut 44 percent in single stream evaluation.

The goal is to keep the capability to do it on their own

Scoring well in each round matters, he said, though they are looking further out and trying to do as much as they can under their own power. They tend the whole process themselves, from making the data through to improving the model, and he expects the capability and the know-how built in round two to carry into round three. He hopes the models that come out of it get used widely in Modu's AI, the government-led program that started recently.

A question from someone at a company building AI that understands physics was a good one. A model is software in the end, he said, and anything with a structure can be written as code, and there are people who know how. China publishes recipes and ten thousand papers a year, so why is this so hard? Sungjun Cheon laughed and said he had wanted to brag about this somewhere and had never been given the chance.

Having limited resources is itself a very hard condition, and it pushes you to read a great many papers so you can screen out the bad options, because model development throws up countless choices. The hardest part he named was the width of what has to be covered, since a foundation model carries pressure to be good at nearly every capability. Gathering physics data alone is hard, he told the questioner, so imagine that across every domain. The people, the resources and the work all multiply, and dividing it up systematically gets hard. Model experts and linguists, service developers and full stack engineers, kernel engineers and specialists in each layer of the training software stack all have to work together.

Another question came about human in the loop. People are a finite resource whose own consistency can waver, so how did they secure accuracy in the evaluation? There are techniques like shuffling the order or taking a majority across several raters, he said, while what matters more is how you compose the data. You build the data first around problems that sit on a genuinely ambiguous boundary, in nuance or in whether an answer is truly bad. Humans do not scale, so they should look only at what is really on the boundary. They put their effort into cutting the numbers down, spinning the loop faster and keeping the stages connected so information moves between them.

Nothing happens in the plant without reading the diagram

The last session was Hongjun Ahn of Trillion Labs. He opened by saying this was his first talk to a community this size. Trillion Labs is a foundation model company visible enough to be listed as Korea's regional AI at NVIDIA GTC, and it has open sourced 7B, 21B and 70B dense models. Opening a 70 billion parameter model together with its mid-training checkpoints was a first in Korea, and Tri-21B-Think, announced earlier this year, sat inside the top 30 of the Artificial Analysis intelligence index. What they are building now is Gravity, an Industry Foundation Model series aimed at AI factories rather than at chat.

The future he ultimately draws was the spine of the talk, an AI agent that can genuinely self-drive complex physical infrastructure, with a bird's eye view of the facility as it investigates an incident, reasons about the cause and coordinates the response. The model has four kinds of tasks to solve. It reads a P&ID, the very large diagram of how modules connect, to understand the outline of the whole infrastructure. It understands data from a great many sensors so a failure never blows up. It understands the strict manuals and complicated module documents for when something does go wrong. And it understands the long time series the sensors have piled up.

They are deploying it at a 1.2 gigawatt thermal power plant. The copilot has to understand a complicated system, detect a sensor value that suddenly stops arriving or jumps, find the problem, and then go past that to recommend the action that fixes it. To define the problem in the first place they built an internal benchmark called PowerBench, since nothing on the shelf covered it. The P&ID is the ground truth of that plant, questions from the field get answered on the diagram, and the manuals and the text do not hold the answer.

If you cannot read the diagram, you cannot act in the plant.

The tasks split three ways. Connectivity and topology asks things like which valve to close to isolate P-101B from the discharge header. Reading symbols and tags asks what the relief valve tag on the reflux drum is. Safety and process logic asks which interlock fires at the high high level on V-201. Every one of those is a question an engineer on shift would ask out loud. An LLM judge scores the answers against annotated ground truth, and that score becomes the reward in reinforcement learning directly, with no separate reward model in between.

Reward never reaches a policy that cannot read

Why the pipeline runs in that order was the clearest thing I heard all day. A base VLM has never seen P&ID conventions, and reinforcement learning only sharpens what a policy can already sample, so rollouts from a policy that cannot read the diagram never reach the reward. They raise the prior first and run reinforcement learning after. Mid-training takes full resolution diagrams and crops of them, standards and engineering documents on reading a diagram, and a QA set grounded in diagrams. SFT runs in two layers, QA format SFT for answer style and output format, agent trajectory SFT for the zoom traces that cold start reinforcement learning. Symbols, then format and the tool loop, then where to look.

He admitted he had wondered at first why a harness needed designing at all. Just hand the model the diagram and let it solve the thing, he thought. Then he scraped together real diagram data and found sheets at roughly A0 size and 16 megapixels, and feeding something that large in one piece takes a long while on prefill alone. Downscaling to 4 megapixels fits inside the budget, while local information like tags and symbols blurs and then disappears outright, which is exactly the information the question was about.

So they kept the whole view and let the model zoom when it needs to. The action space has zoom, which re-renders a region at native resolution and returns a readable crop, and answer, which ends the episode, and the model can zoom again inside a crop. When a query arrives asking what has gone wrong and what to do about it, the model zooms in sequence, works its way toward the element where the problem probably sits, and produces an answer. Episodes run to variable length as the agent decides where and how often to look, and reinforcement learning runs over the whole trajectory so the policy learns where it should be looking.

They took GPU utilization from thirty percent to eighty

For large scale reinforcement learning they went asynchronous. Why the synchronous version does not scale was clear enough. Agent rollouts have a long tail, so every step waits on the slowest one and then stops again at weight synchronization, and the trainer idles while rollouts run while the rollout servers idle during training. GPUs ran at around thirty percent, and it got worse as the number of turns grew, which is the wrong direction for a system built on long agent episodes.

In the asynchronous version the workers stream trajectories without pause, the learner trains without waiting, and the only thing that stops the GPUs is weight synchronization. The key number here is how many versions behind the learner a trajectory is allowed to be. Most setups put it at one or two, while they found a recipe that pushes it to eight and still trains stably. Throughput went up more than six times against synchronous, and GPU utilization went from thirty percent to eighty.

Someone asked whether a model might break if you left it at eight and trained for a long time. They had never run two or three thousand steps at a stretch, he said, and nothing had ever broken, which came from shaving the algorithm down rather than from fixing a break they watched happen. With a PPO-based algorithm they dropped the clipping most people use, since training went fine without it. Capping the inference sampling ratio mattered more for off-policy data, and anything about to blow up gets masked out of the gradient. Balancing the trainer against inference mattered too, since the real average sits near three or four. Twelve ran fine internally, though eight wins on wall clock.

Signs that arrive as sound first are still unsolved

The last question touched the furthest point in the talk. Someone from a Sound AI startup noted that the published material mentions design diagrams, time series data and maintenance history as the inputs, and said that in data centers and large power facilities there is a band where a sign shows up as sound or vibration before telemetry numbers ever catch it. How would the world model handle that band, and was there a plan for it at all?

Hongjun Ahn called it a really good question and paused for a moment. He had talked with field engineers a great deal during this project, he said, and realizing the larger vision means building a world model that can eventually predict data like that. What came next was honest.

At this level, honestly, I do not know. The answer may sound lazy, but at this point I really have no feel for it.

Understanding it is probably within reach, he said. You can synthesize problems related to that data one way or another. Finding a more leading indicator, predicting from it and then acting on the prediction is a very hard problem. The answer came from someone who is actually solving the problem, which is why it stayed with me.

The team that builds the table and the teams on it shared a room

We closed with the small group discussion that is the heart of a Bloom event. People form groups of four to six, share their name, where they work and why they came, and then keep the conversation going. The biggest laugh came from how we pick the group leader. Whoever spends the most of their own money on AI leads, and company cards do not count.

The Artificial Analysis team prepared the questions themselves. What is the biggest barrier to adopting and scaling AI in an organization? What matters most for AI progress over the next few years? What new AI use case have you seen lately? And what recent piece of AI news was the most interesting? We asked people to start with whichever one pulled at them, since covering all four was never a requirement. Having the questions come from the visiting team rather than from us changed what the tables talked about.

We asked each group to post its insights in the Discord channel, and people stayed talking well after the event had ended. The team that builds the benchmark, the teams that build the models on that table, and the people who pick among those models were all in one room that night. I was grateful that we could host the first Korean event Artificial Analysis has held, and I hope George can come in person next time, at an hour that suits him better.

Join Bloom

Bloom builds offline rooms where people and technology meet. We hope to see you at the next one.

• 💬 Discord community, where events are announced first
• ▶️ YouTube, full talks from past events
• 📸 Instagram, photos from our events
• 🎟️ Upcoming events