Setting GDALRaster.bands as a cached property was creating a circular reference with objects having __del__ methods, which means the memory could never be freed. Thanks Daniel Wiesmann for the report and test, and Tim Graham for the review.
Thanks to Tim Graham and Claude Paroz for the reviews and patches.