Challenger banks are more exposed to risks and will be more vulnerable in any potential Brexit-caused economic downturn than others, a credit rating agency has warned.