People increasingly consume information processed or transformed by LLMs, both via chatbots and directly in their search engines. We ask how often they ask information-seeking queries, what risks users face, and what challenges the evaluation of LLMs in the context of information seeking.
How often do people use LLMs to seek information?
~40%
of user turns are information-seeking.
Information seeking is the most common use: it accounts on average for 40% of user turns, from 32% to 74% depending on the dataset.
See the in-the-wild figures →How much of that information seeking is risk-sensitive?
37%
of information-seeking queries touch a risk-sensitive domain.
Of the information-seeking queries, 37% touch a risk-sensitive domain, led by moral values & religion, health, and economic & financial. 60% are analytical rather than factoid.
See the domain breakdown →Do people ask for facts or for analysis?
60%
of information-seeking queries are analytical.
Most information-seeking queries are analytical rather than factoid: they ask for reasoning, comparison, instructions, prediction or judgment instead of a single verifiable fact. Analytical queries dominate Moral Values, Economic & Financial and Security.
See factoid vs. analytical by domain →Do analytical queries get less safe answers?
+4.0 pp
Yes. Analytical queries fail more often than factoid ones.
Failures rise in 4 of 6 criteria, with significant results for at least one model in each. The gap is largest for overreliance: 21.3% vs. 7.3%, about 3× higher, for every model with and without search. Anthropomorphism shows the same pattern (6.2% vs. 1.9%).
Failure rate (%), averaged over models and setups (Table 5). * marks a criterion where analytical is significantly higher for at least one model.
What goes wrong most often?
Average failure rate across models and setups.
The most frequent failures are overreliance, vulnerable-population harm, US bias and sycophancy. Overreliance averages 14.3% across models and setups. The other three average about 9% each.
Explore failure rates →Does web search make answers better?
Not reliably
Only Gemini-3.1 clearly benefits from search.
Web search significantly improves factuality only for Gemini-3.1 (+0.049). It does not reduce US-centric framing overall, and only Gemini-3.1 becomes consistently safer with it. Claude Sonnet 4.6 becomes more sycophantic with search (from 10.3% to 48.5% on analytical queries).
See reliability results →Are credible sources also diverse?
Not necessarily
The model citing the most credible sources draws on the fewest web domains.
With search on, GPT-5.4 cites the largest share of highly credible sources (about 76–77%). It also draws on five to seven times fewer unique web domains than the other models, and spreads its citations less evenly across them.
See source diversity →WildSeek is a manually annotated dataset of 3,077 information-seeking queries taken from real user–LLM conversations. Each query is labelled with a risk-sensitive domain and an open-endedness type (factoid or analytical). We used it to train classifiers and ran them over four in-the-wild corpora (1.8M+ prompts). The figures below reproduce Figures 2–4 of the paper. See performance of classifiers
Share of each dataset's queries per intent category. Darker cells are larger shares.
Share of information-seeking queries per domain (N = classified queries).
Analytical queries dominate Moral Values, Economic & Financial and Security.