How is a hobby or celebrity fan site any different than a Pinterest or Tumblr page? You weren't selling prints or using it in marketing material.
No one can tell you what to do, but if it were me, I would firmly make my opinion known that I consider this fair use. I would explain that the entire concept of "fair use" is a rather gray area and they probably disagree with that assessment. However if they decide to pursue this matter I would make it very clear that I would become very proactive and very loud. I would mention friends at the EFF, news media and government agencies. I'd let them know that you are the type of person that would like to establish clarity on this issue and you are willing to go "all the way" to get that clarity.
Something like that.
I followed your advice and Techdirt has posted a story about my site. http://www.techdirt.com/articles/20120515/07173918924/copyright-troll-demands-8500-rarely-visited-lindsay-lohan-fansite.shtml