Martin Paul Eve et al. have published "Cost Estimates of an Open Access Mandate for Monographs in the UK's Third Research Excellence Framework" in Insights.
Here's an excerpt:
The recent ‘Consultation on the second Research Excellence Framework' (REF) in the UK contains an annex that signals the extension of the open access mandate to monographs. In the service of promoting discussion, rather than prescribing a forward route, this article estimates the costs of implementing such a mandate based on REF 2014 volume, taking the criteria signalled in the annex, and identifies funding sources that could support it. We estimate that to publish 75% of anticipated monographic submission output for the next REF would require approximately £96m investment over the census period. This is equivalent to £19.2m per year. Academic library budgets as they are currently apportioned would not support this cost. However, these sums are but a fraction of the total quality-related funding, Arts and Humanities Research Council and Economic and Social Research Council budgets. We close with a series of provocative suggestions for how the mandate could be implemented.