I use 7 differents templates depending on which K2 category you're looking at. Unfortunately, if this is working for Category View, as soon as I click on an item, the item view looses the category link and so switch back to the default template.

For example :
Category View for the "Alimentation" K2 Category is using the template "CpourlesParents"
www.cpour.net/index.php?option=com_k2&view=itemlist&layout=category&task=category&id=97&Itemid=1228

But... Item view for one of the listed items switch back to the default template...
www.cpour.net/index.php?option=com_k2&view=item&id=432:dalloyau-paques-2015&Itemid=1151

I have been looking for days and I'm getting mad :) Any idea ?

The /alimentation/ part of the URL indicates a different menu item.
This means that a) either the menu item's settings are not properly configured or b) the template you are using does not include the item.php file and it reverts to the default one.