A safe bet would be to go with the Silverfish AI at first. You have to check the box here :
As for using own rules or someone elses. You have to UNcheck Silverfish AI, if not your rules will not be used by the bot.
If you want to edit them you will need Excel. Rules can be found in "DeckRuleSet" folder inside HearthRanger's folder. You will have to spend some time reading examples of rules to see how they work and how to edit/create some.
Mulligan rules are easier to create than gameplay rules (in my opinion). I'd start with that.
Best of luck !
Smee