Plans for a town’s first Greggs in a former bank have been rejected due to the “harm” it will have on the conservation area.