I have a category that is accessed through means other than a menu. However, this category displays modules that I don't want it to. Since modules are menu based in how they are published, how would I cancel the modules from being displayed in this category.

You need map the category into a hidden menu item.

I tried that.

1st: I created a hidden menu item and then made sure it wasn't checked for the menu assignment and it didn't work.
2nd: I created a non-hidden submenu item and then made sure to make sure it wasn't checked in the menu assignment and the end result is the same.

When I view the category, I don't see the modules in the category listing. But when I click an item in that category, it does show the modules that I don't want it to.

There is one thing I forgot to mention: the items are also accessible directly from the front page and don't require the menu to access them.

Then you either need to edit the template you are using or you need a 3rd party extension like Modules Anywhere.