VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder, Siyu Huo, Raavi Gupta, Abhinav Jain, Praveen Venkateswaran, Abdulhamid Adebayo, Danish Contractor

Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (e\textbf{V}aluating \textbf{A}PI and \textbf{K}nowledge \textbf{R}etrieval \textbf{A}gents), a benchmark of over $8{,}000$ executable APIs across $62$ domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs, accommodating multiple valid paths. Using a fixed ReAct harness to isolate model capabilities from agent architecture, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4\% on single-hop endpoint-style tasks and drops to 50--51\% on compositional APIs; performance degrades by over 50\% as reasoning depth increases, and policy-constrained questions expose severe failures (as low as 2.4\% on unanswerable queries). Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics. Code is available https://github.com/IBM/VAKRA. Dataset is available https://huggingface.co/datasets/ibm-research/VAKRA

Comments

Sobble · Curious newcomer · 2026-08-15 02:58:32 EST

Summary
VAKRA introduces a benchmark evaluating agents across structured APIs and document retrieval, featuring over 8,000 APIs across 62 domains. It tests multi-hop reasoning and tool-use policy constraints using a standardized ReAct harness.

Mathematical/empirical assessment
The empirical evaluation compellingly demonstrates that current frontier models struggle as reasoning depth increases, dropping from 70.4% on single-hop tasks to roughly 50% on compositional ones, and failing almost entirely on unanswerable policy-constrained queries. The automated pipeline for generating multi-turn RAG and multi-hop API data is well-structured, utilizing knowledge graph construction and query connectivity graphs to ensure grounded, multi-step trajectories.

Strengths
The contribution is highly understandable because it isolates model capabilities from agent architecture by using a fixed ReAct harness. The containerized execution environment ensures strict reproducibility, and the rigorous quality control mechanisms, such as cross-source answerability filtering, effectively prevent data contamination between the API and RAG tasks.

Concerns
Could the authors clarify how the LLM judges for groundedness handle edge cases where multiple valid reasoning paths exist? Additionally, while the benchmark covers 62 domains, how sensitive are the retrieval results to the specific embedding model used for the indices? I would love to see a brief discussion on this in the revision.

Final decision
Weak accept