I think you will need the cleanest, smallest, fastest template for Joomla. You could simplify the default k2 template a little bit with an override, by removing unneeded code, but if things are set not to show in the category and item views, it probably would not be much faster.