Benchmarks

Existing AI benchmarks measure intelligence: reasoning, knowledge retrieval, code generation. We are building something complementary. Taste-Bench measures the qualities a model needs to help with work that runs on judgment, not just correctness. This includes both a model developing its own sense of taste in a domain (acting as a credible critic or advisor) and its ability to support people in their creative practice without flattening what makes their work theirs.

This is not one benchmark but a series, each targeting a different facet of what taste looks like in practice. Some evals resemble games: the model navigates a structured scenario and we measure the quality of its choices. Others ask the model to produce an output (a critique, a set of questions, a ranked list) and use a separate evaluation layer to judge whether the output is good. Each benchmark develops its own methodology fitted to what it is trying to measure.

We are developing a large number of these tests. The plan is to then compare results against all major existing benchmarks and identify which of our evaluations are strictly orthogonal to measures of raw intelligence. The goal is a small, stable set of benchmarks that capture something genuinely new about what models can and cannot do.