Large-scale area-based conservation measures affect millions of people globally. Understanding their social impacts is necessary to improve effectiveness and minimize negative consequences. However, quantifying the impacts of conservation measures that affect large geographic areas and diverse peoples is expensive and methodologically challenging, particularly because such evaluations should capture locally defined conceptions of well-being while permitting policy-relevant comparisons. Here, we measure the impact of Tanzania’s Wildlife Management Areas (WMAs), a national community-based conservation and poverty reduction initiative. We use a novel, cost-effective impact evaluation method based on participatory wealth ranking and Bayesian multilevel modelling. We find that from 2007 to 2015 the impacts of WMAs on wealth were small and variable, with no clear evidence of widespread poverty reduction. Accompanying qualitative data suggest that apparently positive effects in one WMA cannot be directly attributed to WMA activities. Our results suggest that current WMA policy needs to be revisited if it is to promote positive local development.