WildSeek is a manually annotated dataset of 3,077 information-seeking queries taken from real user–LLM conversations. Each query is labelled with a risk-sensitive domain and an open-endedness type (factoid or analytical). We used it to train classifiers and ran them over 1.8M+ real queries. Alongside the dataset, we propose an evaluation framework for LLMs in the context of information seeking, which scores responses for reliability, fairness and safety. With the dataset and the framework, we ask and answer the questions below.
How often do people use LLMs to seek information?
~40%
of user turns are information-seeking.
Information seeking is the most common use: it accounts for about 40% of user turns, from 42% in LMSYS-Chat-1M to 74% in SES.
See the in-the-wild figures →How much of that information seeking is risk-sensitive?
37%
of information-seeking queries touch a risk-sensitive domain.
Of the information-seeking queries, 37% touch a risk-sensitive domain, led by moral values & religion, health, and economic & financial. 60% are analytical rather than factoid.
See the domain breakdown →Do people ask for facts or for analysis?
60%
of information-seeking queries are analytical.
Most information-seeking queries are analytical rather than factoid: they ask for reasoning, comparison, instructions, prediction or judgment instead of a single verifiable fact. Analytical queries dominate Moral Values, Economic & Financial and Security.
See factoid vs. analytical by domain →Do analytical queries get less safe answers?
+4.0 pp
Yes. Analytical queries fail more often than factoid ones.
Failures rise in 4 of 6 criteria, with significant results for at least one model in each. The gap is largest for overreliance: 21.3% vs. 7.3%, about 3× higher, for every model with and without search. Anthropomorphism shows the same pattern (6.2% vs. 1.9%).
Failure rate (%), averaged over models and setups (Table 5). * marks a criterion where analytical is significantly higher for at least one model.
What goes wrong most often?
Average failure rate across models and setups.
The most frequent failures are overreliance, vulnerable-population harm, US bias and sycophancy. Overreliance averages 14.3% across models and setups. The other three average about 9% each.
Explore failure rates →Does web search make answers better?
Not reliably
Only Gemini-3.1 clearly benefits from search.
Web search significantly improves factuality only for Gemini-3.1 (+0.049). It does not reduce US-centric framing overall, and only Gemini-3.1 becomes consistently safer with it. Claude Sonnet 4.6 becomes more sycophantic with search (from 10.3% to 48.5% on analytical queries).
See reliability results →Are credible sources also diverse?
Not necessarily
The model citing the most credible sources draws on the fewest web domains.
With search on, GPT-5.4 cites the largest share of highly credible sources (about 76–77%). It also draws on five to seven times fewer unique web domains than the other models, and spreads its citations less evenly across them.
See source diversity →These are 3,077 information-seeking queries sampled from four corpora of real human–LLM conversations. Annotators labelled each one by hand with the risk-sensitive domain it touches and whether it asks for a fact or for analysis.
queries in total
Is the turn information-seeking at all? There are five classes: information seeking, content creation, coding, no request, not English.
Browse intent labels →Domains where a wrong or incomplete answer could affect someone's life, safety, health, money or decisions.
Factoid queries can be answered from a single verifiable source. Analytical queries need reasoning, comparison, instructions, prediction or judgment.
Our ModernBERT classifiers labelled every turn in four in-the-wild corpora (1.8M+ prompts). The figures below reproduce Figures 2–4 of the paper. See performance of classifiers
Share of each dataset's queries per intent category. Darker cells are larger shares.
Share of information-seeking queries per domain (N = classified queries).
Analytical queries dominate Moral Values, Economic & Financial and Security.