29 January 2026
Over the past few years, repositories have been encountering a growing number of bots trying to access their resources. These bots, or crawlers, navigate the internet, gathering data and indexing information for search engines and large language models, as well as for other purposes. While some of these bots are rather innocuous, others are sufficiently aggressive that they cause service disruptions in repositories (and other scholarly communications infrastructures).
Machine users have always been a critical constituency for repositories as search engines, aggregators, and other indexing services represent the predominant mechanisms whereby a repository’s resources are discovered. As such, it is very much in the community’s interest to ensure that repositories remain open and accessible to friendly bots and crawlers despite the increasingly aggressive nature of some bots.
In early 2025, COAR conducted a survey to assess the extent to which repositories were being adversely affected by a sudden and large increase in activity by “bots” – automated Web clients such as crawling or harvesting systems. The ensuing report – The impact of AI bots and crawlers on open repositories was published in June 2025. It makes clear that:
- the scale of traffic from badly-behaved bots presents a significant problem for open-access repositories
- the measures being taken by repositories vary in their effectiveness
- some of the measures being taken by repositories have unintended consequences of impeding access from legitimate users (both human and “machine”)
In response to this, COAR convened the Dealing With Bots Task Group to develop advice and supporting information for repository managers to help them to deal with this phenomenon.
The primary output of the Task Group is a Dealing with Bots website, which is intended to help repository managers make informed decisions about strategies they might use to achieve this balance in their own context.
For more, see: