Publish down is used for scheduling an item to be unpublished. It does not save WHEN the item was manually published.

For that you'd need a custom plugin that would mark the date an item was unpublished in a separate database table in your site. If you're interested, we could develop this for you (contact us via www.joomlaworks.net/support/contact) or you could ask another 3rd party extension developer from getk2.org/extend.