To answer this question, you need to understand the consequences of leaving or removing plant growth. This growth on brick can potentially damage it by forcing root tendrils into the mortar joints. Moisture can then find its way into the wall and freeze-thaw action or other moisture related events can occur resulting in damage. However, ivy also sheds rainwater and reduces the surface temperature of the wall. Properly constructed walls with good workmanship and well-tooled joints can also resist tendril intrusion better than poorly constructed walls.
When existing plant growth is removed from a brick wall, it may also remove part of the mortar from the joint. This could result in the wall having to be repointed by removing any broken mortar and repacking with new. Further, complete removal of the tendrils is difficult. Failure to do so may result in stains on the wall.
Therefore, it is an evaluation which is best made by the owners taking these as well as the aesthetic and economic considerations into account. While plant growth can shorten the life of brickwork, a well-constructed wall should still last for many decades.
Additional information on plant growth on brickwork is available by ordering Engineering and Research Digest #621 from the Brick Bookstore.