On July 26, 2024, NIST released AI Risk Management Framework documents, including a Generative AI Profile.

TL;DR

• This article examines agents that help users construct preferences, not only elicit existing ones.
• This matters because better support can aid decisions, but stronger steering can raise manipulation risks.
• Review agent flows, expand evaluation beyond accuracy, and add consent-aware explanation steps.

Example: A user compares unfamiliar options and cannot explain a preference yet. The agent first clarifies tradeoffs, then helps the user form priorities.

The EU AI Act, enacted in 2024, drew a line against manipulative, deceptive, or vulnerability-exploiting AI practices. A paper posted to arXiv in June 2026 highlights a gap between that policy context and current agent design. Many agents are built to ask for user preferences. The paper argues that, when users do not yet know what they want, agents should help with preference construction.

TL;DR

• The core issue is whether agents remain tools for eliciting formed preferences, or help users construct preferences.
• This shift is not only a UX issue. It can support non-experts, but it can also increase steering risks.
• Agent evaluation should extend beyond accuracy. It should examine presentation, user effort, consent-based intervention, and decision quality.

Current status

The problem statement is clear. The arXiv abstract says agents often assume expert users with well-formed preferences. When tasks are less specific, agents often respond with clarification questions. The abstract argues that this assumption is unrealistic. Users may be unable to answer preference questions without domain knowledge.

This point connects to limits in existing evaluation frameworks. KnowU-Bench is a mobile agent evaluation task released on arXiv in 2026. According to the findings, it does not sufficiently test several issues. These include eliciting missing preferences through interaction. They also include when to intervene, remain silent, or ask for consent. Standard evaluation still focuses more on inferred intent than joint refinement of intent.

UX research also offers clues. A systematic literature review on human-AI collaboration treats on-demand AI help as an important interaction pattern. Research on explainability proposes open, conversational explanations instead of one-off messages. Earlier studies on preference elicitation interfaces also noted possible misalignment. The system’s quantitative representation can differ from the user’s internal representation. If a system asks only through numeric sliders, users may answer in the system’s format.

Analysis

This issue matters because it challenges basic assumptions in agent design. Many agent UX patterns have centered on asking more questions when requests are ambiguous. That approach can break down for non-experts facing unfamiliar problems. Examples include insurance products, degree programs, medical information, financial options, and complex software purchases. Users may lack the language needed to answer direct preference questions.

In those cases, a different flow may be better. The agent can first explain comparison frames, key concepts, and possible side effects of each choice. It can then ask which dimensions matter more. As agents move from search boxes to copilots, alignment expectations broaden. The goal is not only instruction following. The goal also includes helping understanding without excessive steering.

Safety concerns emerge at the same point. Helping construct preferences means intervening in how users form judgments. This is why the OECD AI Principles emphasize autonomy, dignity, and understandable information. NIST places weight on measurement, evaluation, standards, and guidelines. The EU AI Act prohibits manipulative, deceptive, and vulnerability-exploiting practices. A 2025 study also warned that human decision-making may be vulnerable to AI-based manipulation. Preference-construction agents can be more helpful. They can also become more capable persuasion systems. The boundary depends less on the recommendation itself. It depends more on comparison methods, hidden information, and stopping behavior.

Practical application

If a team wants to use this concept in a product, the first design principle should be support for understanding. Immediate answers may be less useful when users cannot answer direct questions. The system should first explain dimensions that separate the options. It can then ask which dimension matters more. A purchasing agent, for example, should not jump directly to a final recommendation. It should present criteria such as price, maintenance cost, learning difficulty, and compatibility. It should also let users revise those criteria. Explanations should be conversational, not one-time notices.

Evaluation should also change. Final recommendation accuracy alone can mislead teams. An agent that pressures users into quick answers may appear stronger on that metric. Evaluation should jointly examine multi-turn tasks with hidden latent preferences. It should also assess information quality, comparison quality, question strategy, user effort, restraint after refusal, and consent-based intervention. These items align with benchmark design directions in the findings.

Checklist for Today:

• Find consecutive clarification-question sections and add a concept explanation plus a comparison-frame step before them.
• Add user effort, explanation comprehension, and non-consensual intervention frequency to the offline evaluation rubric.
• Add text on each recommendation screen explaining why a criterion appeared first and how to revisit options.

FAQ

Q. Is a preference-construction agent just a system that gives more recommendations?

No. The core is not the quantity of recommendations. The core is helping users understand decision criteria they had not recognized. It also helps them set their own priorities. The system should provide information, comparison, explanation, and revisable criteria.

Q. If this approach is adopted, does personalization improve?

Possibly. Based on the findings alone, we cannot conclude consistent improvement over conventional clarification-question UX. An evaluation design tailored to domains and user groups should come first.

Q. What is the biggest risk?

Excessive steering. Under the pretext of helping users understand, a system may push a particular choice. It may also unsettle the judgment of vulnerable users. That is why evidence-based explanation, user control, human oversight, and consent-based intervention matter.

Conclusion

A preference-construction agent is not only a debate about a smarter chatbot. It is a reconsideration of agent design assumptions. Many systems have assumed users already know their answers. Future differentiation may depend less on saying more. It may depend more on helping users build preferences more fairly.

Further Reading

• AI Employment Narrative Shifts From Loss to Redesign
• AI Resource Roundup (24h) - 2026-07-01
• Why Generator Evaluator Consistency Matters In LLM Self-Review
• When Public AI Transparency Becomes Mere Paper Compliance
• Rethinking AI Tutors Beyond Cloud Chatbots in Education

References

• AI principles | OECD - oecd.org
• Artificial intelligence | NIST - nist.gov
• AI Risk Management Framework | NIST - nist.gov
• AI Act | Shaping Europe’s digital future - digital-strategy.ec.europa.eu
• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation - arxiv.org
• Decisive: Guiding User Decisions with Optimal Preference Elicitation from Unstructured Documents - arxiv.org
• Preference Construction: A Bayesian Interactive Preference Elicitation Framework Based on Monte Carlo Tree Search - arxiv.org
• Human Decision-making is Susceptible to AI-driven Manipulation - arxiv.org
• Human-AI collaboration is not very collaborative yet: a taxonomy of interaction patterns in AI-assisted decision making from a systematic review - frontiersin.org
• Artificial agents’ explainability to support trust: considerations on timing and context - link.springer.com
• Designing interfaces for explicit preference elicitation: a user-centered investigation of preference representation and elicitation process - link.springer.com
• arxiv.org - arxiv.org