Brokers have been urged to branch out into specialist lending as a way to diversify against falls in mortgage activity.