[GTALUG] Ubuntu (or debian): apt-get auto-remove?

After my update from Ubuntu 12.04 to 14.04, apt-get tells me that there are a lot of packages that apt-get auto-remove could remove. Is that safe? What is the underlying idea? I think that it removes things that were added due to dependencies where those dependencies no longer exist. In the back of my mind, I worry that sometimes dependencies might be things that I wanted independently of why they were installed, but I'm not sure of that. Do people fine apt-get auto-remove works great or do you find that it throws a few babies out with the bathwater?