I finally have my playbook and rules in place. Now I want to know how to best evaluate my plays and their effectiveness.
How many times should I allow an offensive play or defensive set to be run before deciding whether to retain it or throw it out? Is 10 enough? 15? I don't want to react too quickly to a small sample size, but I also don't want to hurt my team by moving too slowly and letting them run a play/set that is never going to work.

That's a tough question.

It takes soooooo long for players to learn a new play, it's hard to know if the play doesn't work because it's a bad play, or because your team doesn't know it well enough yet.

I would normally wait at least half a season before I give up on a play.

FYI - Don't even bother trying any of the Shotgun running plays.