so I have about 100 servers running 16.04 spread over North America. Today it became apparent that the root user cron script was deleted on all 100 of them. The script is at /var/spool/cron/crontab/root and everything I had has been deleted and replaced with # DO NOT EDIT THIS FILE - edit the master and reinstall. # (- installed on Fri Dec 28 08:18:31 2018) # (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $) crontabs were all originally installed using the crontab executable. when I redo it the same way I get the proper crontab file but with this above header added. This was never the case before. I do not have automatic updates turned on on any of my machines. The date stamp in this header is within a few seconds of the above on all my machines. I also have a few machines running Armbian, and they are exactly the same. This is a major F*up for me and I have no idea how it happened. Has anybody else had similar experiences? I'd love to know. regards, darryl
I know that crontab -r removes the user's crontab, but what's more likely (based on your content) is that a new version of cron was installed -- and that process somehow overwrote the existing crontab with what looks like a default version. I have a line in my crontab that does a periodic save: # 2018-1121: 1533: Save the current crontab for later backup 36 8,15 * * * crontab -l >/home/web/crontab.latest I then use rsync to back that file (and others) up to a safe place. Alex On Wed, Jan 9, 2019 at 11:45 AM Darryl Moore via talk <talk@gtalug.org> wrote:
so I have about 100 servers running 16.04 spread over North America.
Today it became apparent that the root user cron script was deleted on all 100 of them. The script is at /var/spool/cron/crontab/root and everything I had has been deleted and replaced with
# DO NOT EDIT THIS FILE - edit the master and reinstall. # (- installed on Fri Dec 28 08:18:31 2018) # (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
crontabs were all originally installed using the crontab executable. when I redo it the same way I get the proper crontab file but with this above header added. This was never the case before. I do not have automatic updates turned on on any of my machines. The date stamp in this header is within a few seconds of the above on all my machines. I also have a few machines running Armbian, and they are exactly the same.
This is a major F*up for me and I have no idea how it happened. Has anybody else had similar experiences? I'd love to know.
regards, darryl --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
-- Alex Beamish Software Developer / https://ca.linkedin.com/in/alex-beamish-5111ba3 Speaker Wrangler / Toronto Perlmongers / http://to.pm.org/ Baritone, Operations Manager / Toronto Northern Lights, 2013 Champions / www.northernlightschorus.com Certified Contest Administrator / Barbershop Harmony Society / www.barbershop.org
darryl, you should be able to look at yum or apt/dpk histories to see if/when cron was updated and possibly gleam some information about who/what did it. for debian and ubuntu : https://serverfault.com/questions/175504/how-do-i-get-the-history-of-apt-get... for redhat/rpm based distros: 'yum history info crontab' or whatever the name of the cron package is called Certainly sounds like something automatically updated the cron package to me. Good luck in the forensics. -jason On Wed, Jan 9, 2019 at 12:30 PM Alex Beamish via talk <talk@gtalug.org> wrote:
I know that crontab -r removes the user's crontab, but what's more likely (based on your content) is that a new version of cron was installed -- and that process somehow overwrote the existing crontab with what looks like a default version.
I have a line in my crontab that does a periodic save:
# 2018-1121: 1533: Save the current crontab for later backup 36 8,15 * * * crontab -l >/home/web/crontab.latest
I then use rsync to back that file (and others) up to a safe place.
Alex
On Wed, Jan 9, 2019 at 11:45 AM Darryl Moore via talk <talk@gtalug.org> wrote:
so I have about 100 servers running 16.04 spread over North America.
Today it became apparent that the root user cron script was deleted on all 100 of them. The script is at /var/spool/cron/crontab/root and everything I had has been deleted and replaced with
# DO NOT EDIT THIS FILE - edit the master and reinstall. # (- installed on Fri Dec 28 08:18:31 2018) # (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
crontabs were all originally installed using the crontab executable. when I redo it the same way I get the proper crontab file but with this above header added. This was never the case before. I do not have automatic updates turned on on any of my machines. The date stamp in this header is within a few seconds of the above on all my machines. I also have a few machines running Armbian, and they are exactly the same.
This is a major F*up for me and I have no idea how it happened. Has anybody else had similar experiences? I'd love to know.
regards, darryl --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
-- Alex Beamish
Software Developer / https://ca.linkedin.com/in/alex-beamish-5111ba3 Speaker Wrangler / Toronto Perlmongers / http://to.pm.org/ Baritone, Operations Manager / Toronto Northern Lights, 2013 Champions / www.northernlightschorus.com Certified Contest Administrator / Barbershop Harmony Society / www.barbershop.org
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
On 1/9/19 12:46 PM, Jason Shaw via talk wrote:
darryl, you should be able to look at yum or apt/dpk histories to see if/when cron was updated and possibly gleam some information about who/what did it.
for debian and ubuntu : https://serverfault.com/questions/175504/how-do-i-get-the-history-of-apt-get... <snip
Certainly sounds like something automatically updated the cron package to me. Good luck in the forensics.
Sounds bad on all counts. I'm not aware of any bugs in 16.04 that would wipe out any crontabs on auto-updates or manual updates. If you're the lucky person to discover one, it will definitely require an SRU update to the cron package itself. Does anything show up related to cron in /var/log/apt/*.log as Jason pointed out? What about in syslog and auth.log? Anything there that would show something like 'crontab -r' being invoked? Lastly, are you using any config management tool like puppet, chef, salt, ansible, juju, etc.? My immediate reaction upon reading this is to cast aspersions at config management - think sorcerer's apprentice and all that. Cheers, Jamon
On Thu, 10 Jan 2019 at 13:17, Jamon Camisso via talk <talk@gtalug.org> wrote:
On 1/9/19 12:46 PM, Jason Shaw via talk wrote:
darryl, you should be able to look at yum or apt/dpk histories to see if/when cron was updated and possibly gleam some information about who/what did it.
for debian and ubuntu :
https://serverfault.com/questions/175504/how-do-i-get-the-history-of-apt-get... <snip
Certainly sounds like something automatically updated the cron package to me. Good luck in the forensics.
Sounds bad on all counts. I'm not aware of any bugs in 16.04 that would wipe out any crontabs on auto-updates or manual updates. If you're the lucky person to discover one, it will definitely require an SRU update to the cron package itself.
Does anything show up related to cron in /var/log/apt/*.log as Jason pointed out?
What about in syslog and auth.log? Anything there that would show something like 'crontab -r' being invoked?
Lastly, are you using any config management tool like puppet, chef, salt, ansible, juju, etc.? My immediate reaction upon reading this is to cast aspersions at config management - think sorcerer's apprentice and all that.
Just because you can't control the brooms doesn't mean we can't control the brooms! ... Oh, wait, yeah. They do some pretty wonky things occasionally. I'm forced to acknowledge that you should probably check the brooms if you use them ... If that's the case though, their traces would still normally show up in the logs previously mentioned. They would with Ansible anyway. [for those not following this overstretched metaphor, "brooms" == "config management tools"] -- Giles https://www.gilesorr.com/ gilesorr@gmail.com
participants (5)
-
Alex Beamish -
Darryl Moore -
Giles Orr -
Jamon Camisso -
Jason Shaw