UrbanDS and the data discovery problem in LLM agents

The practical conclusion from UrbanDS is that the core system design problem should be framed less as “making LLM agents write better analysis code” and more as “deciding which data should be found and used.” In organizations such as urban planning, the public sector, finance, and healthcare, data is often scattered across repositories and stored in different formats. In that setting, giving a single agent a few files and asking it to perform analysis can become a bottleneck. UrbanDS studies this bottleneck through a graph-based design and separation of agent roles.

The issue is not model capability, but the approach to data access

A common assumption in existing LLM data science agents is that “the dataset to be analyzed is already provided.” Under this assumption, the agent’s ability to generate code, fix errors, create charts, and train models becomes the main factor. Real urban tasks can be different. Relevant data may not be contained in a single table; it can span multiple institutions, formats, spatial units, and temporal units.

This is also the limitation identified in the UrbanDS paper’s abstract. According to the paper, existing methods tend to rely on limited provided datasets and struggle in data-intensive situations that require discovering and using relevant information from large-scale, heterogeneous repositories. The authors present urban tasks as a representative example of such situations.

This distinction has direct implications for product design. In an internal analytics environment where data has already been organized, code-executing agents may be efficient. By contrast, when the data catalog is large, relationships between datasets are unclear, and required sources vary by task, a layer for data discovery, selection, and relationship understanding may be needed before an analysis agent.

UrbanDS’s core mechanism: the dataset graph

UrbanDS’s graph guide organizes reusable skills for each dataset and relationships between datasets into an integrated dataset graph. The important point is that the graph is not presented merely as an index or search list. According to the paper, it serves as a structural hint that helps the agent find task-relevant data and formulate a plan.

At runtime, the Planner Agent searches the graph, selects relevant datasets, and creates an execution plan. The Relation Agent identifies relationships between datasets and integrates them into the graph. In other words, UrbanDS is not a simple batch-call workflow of “question → the LLM selects data on its own → analysis.” It is closer to an approach that turns a data repository into an explorable knowledge structure and plans on top of it.

The proposed advantage of this design is reusability. Once dataset handling knowledge and dataset relationships are structured, that knowledge can be reused in later tasks. In large-scale repositories, this matters. Adding long data descriptions to every prompt or relying on a person to manually select relevant tables becomes more costly as the number of tasks grows.

There are also limitations. Based on the available evidence, it is difficult to verify how retrieved datasets are physically joined, merged, or transformed in practice. UrbanDS should therefore not be read as showing that “heterogeneous data can now be automatically merged.” The graph may help with data discovery and planning, but the accuracy of actual integration, key matching, unit conversion, and quality validation remains a separate matter for verification.

Performance numbers are a signal, not a basis for purchase decisions

According to the authors’ report, UrbanDS recorded 70.0% overall accuracy on UrbanDS-Bench. This was higher than Claude Code, which had the highest score among the comparison targets at 62.9%. The paper presents this as an 11.2% relative improvement. On CoDA-Bench, UrbanDS was reported at 46.2%, while Claude Code was reported at 42.0%. The comparison targets included data science agents such as DS-Agent, Data Interpreter, and DeepAnalyze, as well as general-purpose agents such as AutoGen and Claude Code.

These numbers support a cautious interpretation: general-purpose agents alone may be insufficient for some data-intensive tasks. They also suggest that a dataset graph and role-separated agent architecture may improve task accuracy.

The interpretation should remain conservative. These are author-reported results from an arXiv preprint. Independent reproduction and statistical significance testing have not been confirmed. The conclusion supported by the current evidence is not “UrbanDS is often better.” It is closer to: in large-scale, heterogeneous data repositories, agents equipped with a graph-based data discovery layer may have a comparative advantage.

Decision rule: look at “relationship uncertainty,” not the number of datasets

When deciding whether to apply an UrbanDS-style approach in practice, the absolute number of datasets is not enough. The key question is whether the data relationships needed for tasks have been defined in advance and remain stable.

If an analytics mart is already well organized, the tables used for each task are mostly fixed, and join rules are clear, an UrbanDS-style graph multi-agent structure may be excessive. In that case, it may be more direct to improve existing data catalogs, pipelines, and code-executing agents.

By contrast, an UrbanDS-style approach is worth considering when several conditions overlap: required datasets vary by task; data descriptions and usage know-how remain tacit knowledge; relationships between datasets are undocumented or change frequently; and analysis failures are caused more by incorrect data selection than by modeling. In such environments, a layer that manages data discovery, selection, and relationship inference may reduce bottlenecks more effectively than further improving the coding ability of LLM agents.

Expansion to finance, healthcare, and public data is possible in principle. The core of UrbanDS is not the urban domain itself, but the structure of creating dataset profiles and relationship graphs in large-scale, heterogeneous repositories and having planning and execution agents use them. In healthcare, cases combining multiple data sources with multi-agent approaches have also been reported. However, there is no confirmed evidence that UrbanDS has directly validated performance on finance, healthcare, or public data. Moving to another domain would require redesigning schemas, access controls, regulations, and evaluation frameworks.

A minimal experiment that can be built now

Rather than trying to replicate all of UrbanDS, an organization can first validate a simpler question internally. “Does graph-based data discovery improve task accuracy compared with human-selected datasets or simple search?”

To examine this, select several representative tasks and compare three approaches against a repository with many potentially relevant datasets. First, a person specifies the relevant data. Second, only text search or existing catalog search is used. Third, dataset descriptions and relationships are structured as a graph, and the agent selects from that graph. Evaluation should not be limited to final-answer accuracy. It should also include the appropriateness of selected datasets, use of unnecessary data, incorrect relationship assumptions, and rerunability.

If the graph-based approach does not perform better in this experiment, there is little reason to scale up to a multi-agent structure. If data selection errors decrease and reusable relationship knowledge accumulates, there is then a stronger basis for expanding role separation, such as adding a Planner Agent or Relation Agent.

Aionda