Broadcaster Carol Vorderman was briefly refused entry to the Wales Office while en route to a Downing Street event when a security guard didn't recognise her, according to a former Welsh secretary.