The only negative is that it destroys your honor score, but you can just play as a vassal of a faction which mitigates that fairly well. So not only is this a great way to weaken a faction, make yourself rich and also grind renown, but the only thing you have to worry about are enemy bandit spawns, which might run from you if your small group of troops are really high leveled. Winning a 5 vs 61 battle will always give you the max 50 renown, and the amount is only lowered once your renown gets really high. Not only that, but the game gives you more renown for winning fights with very few troops against large forces. Lords will be forced to stand and watch helplessly as you sack their villages and they can't do anything about it. Once you kill them all, you're free to raid the village, but this DOESN'T cause you to lose reputation with their faction. You can demand the villages give you free stuff and if you have under a certain number of troops, they'll instead decide to fight you.