Of course WOS smart search could be using a LLM for NLP parsing and NER but they state specifically they using LLM for language translation (non english queries) and then the NLP parser takes over. Also their search query generated is deterministic (likely not LLM) clarivate.com/academia-gov...

Primo NLS seems to expand with synoymns, wos smart search doesnt? But dispite similarity in function, Primo NLS is confirmed as using a LLM but web of science smart search isn't (at least not for this function) & uses "NLP parser" & NER knowledge.exlibrisgroup.com/Primo/Produc... . (3)

Both can take your query input in natural language and within limits, interprete and generate a boolean/field search. In the example below, both correctly limit to 2020 onwards. They can both interpret requests for papers by author etc. Sure there are diffs in terms of fields they can use etc (2)

Trying to build an agentic skill in codex/claude code reminds me of the old Asimov story "Profession". It takes a ton of skill to build a top notch agentic skill! A mix of domain expertise and technical expertise in coding, agentic & context engineering. I suspect currently is very much an art

Scenario 8 is just agentic LLM + tool access to document retriever, beats Scenario 7 - agentic LLM + query reformatter, doc retriever, reranker and even Scenario 9 - agentic LLM + doc retriever (vector similarity) AND graph search

The section on agentic RAG is where the results are most interesting to me, they generally show the superiority of the simple approach of just tell the LLM it has access to a few well chosen tools and let it decide how to iterate and do the work rather than try to over engineer pipelines

Scenario 5 Graph RAG + vector similarity obviously does best for the scenarios under graphRAG but it does not seem much better than just vector similarity + LLM. Of the 5 scenarios the best seems to be the simpler scenario 2 - vector similarity of doc (includng 1 hop relations) & not hybrid RAG

Knowledge graph alone (without LLM generation) do badly which I expected. But even then you can notice, scenario 4 with computed/generate KG does worse than predefined. Adding LLM to answer pushes the results much higher but still lower than just normal vector similarity+LLM.

Looking at the Regular RAG variants I woud say there is not much surprise. Using LLM to generate answer definitely adds a ton of performance than just vector similarity (recall does fall for obvious reasons as LLM does precision-recall tradeoff). Adding into the doc, one-hop relations also helps.

Just put the results here for all 9 scenariosn scored by Hit@1/ Hit@5, Recall@20 and Mean Reciprocal Rank (MRR)

Scenario 6 is dubbed modular RAG - this is what you see in a typical info retrieval pipeline - query reformation->vector search->reranking-> answer generation - this is where most academic ai search tools are/were 2023-2025. 2026 onward you start seeing more agentic systems like in Scenario 7 (6)

They test 2 variants of Regular RAG - Scenario 1 - the text doc only has descriptions of the entity and Scenario 2 includes the entity relationships but in text (4)

in the context of this paper, they tested 3 main versions, 7 scenarios (a) regular RAG - vector similarity of text docs, (b) Graph RAG - uses knowledge graph, (c) Agentic RAG - give the LLM the 2 tools - vector similarity of text docs + graph search and let the LLM decide how to use. (3)

Here's part 2 - this is for the pro-ai librarians. Again don't be offended just tongue in cheek. I can think of many more for both pro and anti AI (2)

A bit of an update of Librarians and their responses to AI. Just tongue in cheek. Do not take too seriously. Generated by chatgpt image model (1)

So I builit one from the scratch reusing the scripts in the old skill that search Pubmed and MesH based on Hausner at.al (2012) - Routine development of objectively derived search strategies - pubmed.ncbi.nlm.nih.gov/22587829/ (2)

Playing around with the Primo MCP server, occasionally recommending librarian expertise

Libkey Nomad browser extend now works on chatgpt, claude web interfaces... This is going to compete with Clarivate's extend browser extension i guess

Vibecoded this using Overton data. aarontaycheehsien.github.io/overtonanaly...

BTW the magic isn't even in the primo MCP. we have a pilot library chatbot implementation that gives it a simple csv file with databases + short descriptions, and even with that the LLM can give just as good answers about databases...(7)

I can go on really, the main magic here is the model itself.. It seems to have at least the level of a entry level academic librarian (thanks to "reading" all the libguides) & has a good sense of what each database includes or not include (6)

a lot of zero search queries are users getting confused and searching in the wrong scope e.g. pasting article citations etc.. No problem the LLM will realise and correct it (5)

Search for autism in databases where no database description will have anything that specific? It reasons and gives Psycino etc... (4)

Asking for Ibisworld or Hoovers, which we don't have, it gives quite good alternatives (3)

Someone searched for "Refinituv" restricted to databases, this is an obvious (if you are into finance) mispelling for Refinitiv which no longer operates as a brand after acquisition. Extra challenge, it is now under LSEG workspace (granted I think our records may have tagged it with refinitiv(2)

“Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation System. - such an interesting paper Explain in detail arxiv.org/pdf/2409.19804

My slide for OCLC Asia Pacific Region in 2018. Not a bad prediction. But not a hard one though at the time I haven't heard of Transformers yet

Figure shows the validation check with and without doi. Its very conservative imho and likely will have false positives but if you want 100% safety... but it does log what it drops so you can check yourself

It does parallel searches of Undermind, Gemini Deep research, Claude own search. Google Scholar (needs api keys), dedupes then does the all important verification check against openalex, crossref, semantic scholar to guard against hallucinations then use opus to screen/ judge relevance

arXiv’s latest warning on AI-generated papers is not just “AI is risky”. The sharper point is author responsibility. If your manuscript contains unchecked LLM output, including hallucinated references, the blame does not shift to the tool. (1)