What Makes Software Good? Making Quality Requirements Measurable

"It works, doesn't it?" Two weeks after go-live, the nightly batch process collapses because ten thousand devices are fetching their data simultaneously. Functionally, everything was correct. Nobody had written down how much load the system needed to handle. This is where the real question of software quality is decided: not just what a system does, but how well it does it — under real-world conditions.

I've talked about this "how well" question in several conference talks. This post distills the core ideas: what quality requirements are, how to make them measurable with quality scenarios, how to prioritize, document, and what generative AI changes about the whole picture.

What Quality Requirements Actually Are

Functional requirements describe what software should do: "Purchasing triggers a reorder." Quality requirements describe how well it does it: fast enough, secure enough, maintainable enough. They define measurable criteria and ensure a system satisfies not just users, but also developers and stakeholders. In short: they are a checklist against expensive rework and avoidable risk.

A solid framework for this is ISO/IEC 25010. It structures software quality into eight top-level characteristics with many sub-characteristics. You don't need to memorize the standard. Use it as a map so no entire quality dimension gets forgotten in a project:

The most important point: quality requirements are more than functional requirements, yet they are the ones most often forgotten. Nobody spontaneously says in a refinement session "by the way, the UI should respond in under 200 milliseconds under load." You have to ask for it actively.

From Vague Wish to Measurable Criterion: Quality Scenarios

"The system should be fast and secure" is not a requirement — it's a wish. It can't be verified, and what can't be verified won't get built. The tool to turn it into something measurable is the quality scenario (from the ATAM method). It describes a concrete, verifiable situation using six building blocks:

• Source: Who or what triggers something?
• Stimulus: What exactly is triggered?
• Environment: Under what conditions?
• Artifacts: Which parts of the system are involved?
• Response: What should happen?
• Measure: How do you know it worked?

An example from an ERP context:

A purchasing employee (source) triggers a reorder to a supplier via the UI (stimulus). It's Monday at 11 a.m. and all employees are using the system (environment). Involved are the UI, database, order service, and warehouse service (artifacts). The system places the order and reports success (response).

Only the measure makes the scenario verifiable — and it covers multiple quality characteristics at once:

• Internationalization: The UI is available in German and English.
• Performance: The UI responds promptly (e.g., < 300 ms).
• Throughput: Order processing completes in time, even under Monday-morning load.
• Security: Employees without the "Purchasing" role cannot place orders.

Each measurable criterion maps directly to a test approach: an integration test for the order flow, a load test for throughput, an authorization test for the role. One scenario, one chain of test criteria.

Prioritizing with the Quality Tree

You can't enforce every requirement with equal rigor — and you shouldn't. A Quality (Attribute Utility) Tree helps you sort: you rate each scenario on two axes, business importance and technical risk or difficulty. What ranks high and is difficult deserves investment. What ranks low and is straightforward gets covered as a side effect.

This prioritization isn't busywork; it's the actual lever. It stops you from building test infrastructure for edge cases while the business-critical core process remains untested.

A Real-World Example

In a consumer goods industry project, the central quality requirement became crystal clear the moment someone asked the right question: throughput of 10,000 devices in one hour at the start of each shift. That's not a vague "must be performant" — it's a number you can test against.

The derived action followed immediately: invest specifically in performance tests optimized for throughput, rather than broad, unfocused testing. One well-formulated quality scenario focused the entire test strategy.

Documenting Quality Requirements

A scenario that only exists in one developer's head is not a requirement. It needs to be findable and ideally machine-readable. Three proven, lightweight formats:

• arc42: a free, standardized architecture template. Quality requirements have a dedicated home there in Chapter 10.
• Architecture Communication Canvas (ACC): a lightweight, visual structure as the baseline for architecture documentation. Rule of thumb: everything else is optional, ACC is mandatory.
• Architecture Decision Records (ADRs): a lean text format that captures concrete decisions and their impact on quality requirements.

The tool matters less than the habit. More or less formal is fine. What matters is that the requirement is documented, versioned, and visible to the team.

Quality in the Age of Generative AI

Generative AI changes this topic in two directions, and you should think about them separately.

AI as a tool. Models help elicit and document requirements, generate machine-readable artifacts, produce diagrams and models, and accelerate iterations. The real gain is a shift in focus: away from writing effort, toward evaluation. A prompt like "Based on this codebase, create an arc42 Architecture Communication Canvas in Markdown, capture key decisions and risks, and include a Mermaid architecture diagram" delivers a first draft in minutes — which you then review and sharpen with domain knowledge.

AI as a system component. Once an AI model is part of your software, new quality requirements emerge: bias, transparency, explainability, and robustness. Existing characteristics like security and performance also gain a new dimension. Your quality scenarios must explicitly cover AI-specific risks; otherwise you're testing a system that behaves fundamentally differently from traditional software.

Do's and Don'ts

From nearly 20 years of project work, here's what actually matters:

Do

• Actively demand quality. With every functional requirement in refinement, ask the follow-up: "How fast, how much data, how secure?"
• Make it measurable. Use quality scenarios or a Quality Tree. A number beats an adjective.
• Question the business domain critically. "How often does this process occur?", "Who uses this feature?", "Are all this data really necessary?" The answers prioritize themselves.
• Focus on core processes and interfaces. Performance and security decisions are usually made at integration points.
• Treat unit and integration tests as the default. They're almost always a good idea and far cheaper than any production incident.
• Select quality requirements deliberately per scenario. Not all characteristics are relevant or valuable for every scenario. Use the list as a guide, not a mandatory checklist.

Don't

• Don't rely on code coverage gates as a quality measure. Once a metric becomes a target, it stops being a good measure (see Goodhart's Law). High coverage can still mean bad tests.
• Don't build heavy cross-system tests too early. They're expensive and slow. Use them sparingly and deliberately, not speculatively.
• Don't copy someone else's test strategy verbatim. The right strategy is project-specific and emerges from your prioritized quality scenarios.
• Don't let tests get in the way. If the test suite blocks or slows every change, it gets bypassed. Then you've missed the point entirely.
• Don't forget AI-specific risks once a model is part of the system.

Quality Is a Shared Responsibility

Good software quality doesn't come from a document or a tool. It comes from a shared attitude, carried from two directions:

• Bottom-up, from development: actively demand, concretize, and machine-readably document quality requirements.
• Top-down, from architecture and management: provide the framework, tools, and backing so quality stays visible, verifiable, and prioritized.

Start small. Take your most important process, write down one quality scenario with a measurable number, and derive a test from it. That's worth more than any standard gathering dust on a shelf. Because in the end: what you don't measure, you can't promise.