Is it possible to force your defense into a certain formation when the offense rolls out in a certain set. The way I understand it now, if an offense comes out in 3WR then the defense matches with nickel automatically. For example, can I set a rule to force my defense to respond to a 3WR set with say a base defense or dime?

Yes. It's an option in Rules.

Well I have it set up but it seems like it still defaults to matching the offensive formation regardless.

In the rules, click on "View Plays" for that rule.

If what you see isn't what you are expecting, there may be something else causing it.

Maybe the general defensive setup in gameplanning (non-rule)?
Or another conflicting rule?

Sorry I can't be much help. I've never gotten too involved in setting up rules because they are too generic.

The engine will choose a play from your rules if one is available before even looking at your game plan, so if your rules are not doing the plays you expect then the first place to look is like WarEagle said and check the actual plays in the rule, and then the second place to look is to make sure the additional conditions are met (score, quarter, time, distance, etc). You can also sort of reverse engineer what actually happened by looking at the play and then figuring out which rule should have applied, a lot of times you'll find a hole in your rule set.

It is more than worth your time to taylor your game plan using rules. For example, you should always call pass defense when facing a 4 or 5 WR set since there are no running plays out of those formations (you'd be surprised at how often that is not the case). Why would you not always call pass defense when defending a 3rd or 4th and long? (Even if you're behind in the score.) More importantly, why wouldn't you always play nickel (or dime) irrespective of the offensive formation in those situations, e.g., against a double TE set?

Although it took quite some time to to so, my entire defensive game plan is predicated on rules. Now that it is set up, I tweek it from time to time depending on what scouting tells me about my next opponent. Wholesale revisions are unnecessary. Then again, I play in only one league.