I'm developing scripts that get run by crontab, so I'm in there making updates fairly regularly. I would love to be able to document the changes, so I'm wondering if there a usual and customary technique to version crontabs? Ideally there would be some sort of hook around 'crontab -e', but failing that, I'd have the output of 'crontab -l' (run regularly by cron?) go to a versioned file. Plan B sounds a bit hokey to me. -- Alex Beamish Software Developer / https://ca.linkedin.com/in/alex-beamish-5111ba3 Speaker Wrangler, Toronto Perlmongers / http://to.pm.org/ Baritone, Board Member, Toronto Northern Lights, 2013 Champions / www.northernlightschorus.com Certified Contest Administrator, Barbershop Harmony Society / www.barbershop.org
I'd recommend using git or a similar version control system, and then have some sort of configuration management tool like ansible put it in place on the server. -jason On Thu, May 3, 2018 at 10:44 AM, Alex Beamish via talk <talk@gtalug.org> wrote:
I'm developing scripts that get run by crontab, so I'm in there making updates fairly regularly. I would love to be able to document the changes, so I'm wondering if there a usual and customary technique to version crontabs?
Ideally there would be some sort of hook around 'crontab -e', but failing that, I'd have the output of 'crontab -l' (run regularly by cron?) go to a versioned file. Plan B sounds a bit hokey to me.
-- Alex Beamish
Software Developer / https://ca.linkedin.com/in/alex-beamish-5111ba3 Speaker Wrangler, Toronto Perlmongers / http://to.pm.org/ Baritone, Board Member, Toronto Northern Lights, 2013 Champions / www.northernlightschorus.com Certified Contest Administrator, Barbershop Harmony Society / www.barbershop.org
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
On Thu, 3 May 2018 at 10:45, Alex Beamish via talk <talk@gtalug.org> wrote:
I'm developing scripts that get run by crontab, so I'm in there making updates fairly regularly. I would love to be able to document the changes, so I'm wondering if there a usual and customary technique to version crontabs?
Ideally there would be some sort of hook around 'crontab -e', but failing that, I'd have the output of 'crontab -l' (run regularly by cron?) go to a versioned file. Plan B sounds a bit hokey to me.
Some years back, I built a bit of infrastructure for this. We didn't have git at the time (heh, AIX was involved. It's always quite the chase to get things running on AIX! ;-) ) I did a combination of push *and* pull on my crontabs: a) I had tooling (integrated using cfengine 2, which tells you how far back! ;-) ) to add desired entries into files. So I used this to modify a copy of the crontab to make sure that the copy included Stuff I Wanted. This represents the "push"; have a way to push things in. But my push was into a *copy*, not the real file... b) Also, nice to have a "pull" If git had been an option, I'd have taken the present contents of crontab, exported via "crontab -l", stow that in git, as a preface to step a). instead, I'd pull the present contents, see if it was different from the latest recorded version, and if so, rename the previous one to mark when it was obsolesced. Then rewrite via a), and use "crontab < authoritative_new_version" to update cron on the new schedule. If one uses an SCM, then that changes a few steps, but not in any huge way. In these modern days, it probably isn't "cfagent+script", but rather "git + ansible/salt/..." -- When confronted by a difficult problem, solve it by reducing it to the question, "How would the Lone Ranger handle this?"
On Thu, May 3, 2018 at 10:44 AM, Alex Beamish via talk <talk@gtalug.org> wrote:
I'm developing scripts that get run by crontab, so I'm in there making updates fairly regularly. I would love to be able to document the changes, so I'm wondering if there a usual and customary technique to version crontabs?
Ideally there would be some sort of hook around 'crontab -e', but failing that, I'd have the output of 'crontab -l' (run regularly by cron?) go to a versioned file. Plan B sounds a bit hokey to me.
I do this using git and SaltStack. Here is my Salt state for managing the root crontab. cron: service.running: - name: cron - enable: True root-crontab: file.managed: - name: /root/root-crontab - source: salt://cronjobs/root-crontab - template: jinja cmd.run: - name: cd /root; /usr/bin/crontab -r; /usr/bin/crontab root-crontab The file called root-crontab is under revision control. One of the things that is in root-crontab is first, updating the Salt states and second, invoking them. That could include updating the root-crontab. There are other ways of doing this in SaltStack. Regards, Clifford Ilkay +1 647-778-8696
Couldn't you do something like cp file file--$(date +%F_%T) ? -- William Park <opengeometry@yahoo.ca> On Thu, May 03, 2018 at 10:44:57AM -0400, Alex Beamish via talk wrote:
I'm developing scripts that get run by crontab, so I'm in there making updates fairly regularly. I would love to be able to document the changes, so I'm wondering if there a usual and customary technique to version crontabs?
Ideally there would be some sort of hook around 'crontab -e', but failing that, I'd have the output of 'crontab -l' (run regularly by cron?) go to a versioned file. Plan B sounds a bit hokey to me.
-- Alex Beamish
Software Developer / https://ca.linkedin.com/in/alex-beamish-5111ba3 Speaker Wrangler, Toronto Perlmongers / http://to.pm.org/ Baritone, Board Member, Toronto Northern Lights, 2013 Champions / www.northernlightschorus.com Certified Contest Administrator, Barbershop Harmony Society / www.barbershop.org
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
On 2018-05-03 11:44 PM, William Park via talk wrote:
Couldn't you do something like cp file file--$(date +%F_%T)
For single file version control you could use the old RCS set of programs. -- Cheers! Kevin. http://www.ve3syb.ca/ | "Nerds make the shiny things that https://www.patreon.com/KevinCozens | distract the mouth-breathers, and | that's why we're powerful" Owner of Elecraft K2 #2172 | #include <disclaimer/favourite> | --Chris Hardwick
On Thu, May 3, 2018 at 11:44 PM, William Park via talk <talk@gtalug.org> wrote:
Couldn't you do something like cp file file--$(date +%F_%T)
You could but that doesn't do anything other than scatter a bunch of files on your disk that are timestamped. It doesn't meet the requirement of documenting changes. A revision control system can do that. The Salt/Ansible/Puppet solution will enable you to deploy the changes from the revision control system. Regards, Clifford Ilkay +1 647-778-8696
On 2018-05-03 10:44 AM, Alex Beamish via talk wrote:
I'm developing scripts that get run by crontab, so I'm in there making updates fairly regularly. I would love to be able to document the changes, so I'm wondering if there a usual and customary technique to version crontabs?
What are you changing in crontabs so much that you need versioning? Shouldn't all the logic be in your scripts? Maybe I'm missing something, but it seems like needless complication to me.
Ideally there would be some sort of hook around 'crontab -e', but failing that, I'd have the output of 'crontab -l' (run regularly by cron?) go to a versioned file. Plan B sounds a bit hokey to me. If you do need to version things, you can just chuck a git repo under /var/spool/cron/crontabs and ignore any files there you don't want tracked.
Cheers, Jamon
On Fri, May 4, 2018 at 8:11 AM, Jamon Camisso via talk <talk@gtalug.org> wrote:
On 2018-05-03 10:44 AM, Alex Beamish via talk wrote:
I'm developing scripts that get run by crontab, so I'm in there making updates fairly regularly. I would love to be able to document the changes, so I'm wondering if there a usual and customary technique to version crontabs?
What are you changing in crontabs so much that you need versioning?
I'm developing EDI scripts, and as development progresses, some of the older scripts are being replaced by newer scripts. On occasion, there are times when I need to disable a crontab job while I solve an issue. Having version control would be a great back-stop, and would also be useful from a historical point of view. (Why did I disable that job last Thursday? Where did the such and such script go?)
Shouldn't all the logic be in your scripts? Maybe I'm missing something, but it seems like needless complication to me.
What to do and how to do it is in the scripts. How often to do it is in the crontab.
Ideally there would be some sort of hook around 'crontab -e', but failing that, I'd have the output of 'crontab -l' (run regularly by cron?) go to a versioned file. Plan B sounds a bit hokey to me. If you do need to version things, you can just chuck a git repo under /var/spool/cron/crontabs and ignore any files there you don't want tracked.
Under my system, that directory is owned by root, and I prefer not to version anything as root; the code I'm developing is application code, not system code. The solution I'm leaning towards is to have a shell script for updating the crontab that would do something like 1. pushd $local_crontab_directory 2. crontab -e 3. crontab -l >crontab.$username 4. git diff crontab.$username If there were changes revealed by the 'git diff', and if I felt they were significant, I could do a git commit, followed by a popd. Thanks for your questions -- they helped me clarify my understanding of the problem and a possible solution. -- Alex Beamish Software Developer / https://ca.linkedin.com/in/alex-beamish-5111ba3 Speaker Wrangler, Toronto Perlmongers / http://to.pm.org/ Baritone, Board Member, Toronto Northern Lights, 2013 Champions / www.northernlightschorus.com Certified Contest Administrator, Barbershop Harmony Society / www.barbershop.org
On 04/05/18 09:10 AM, Alex Beamish via talk wrote:
On Fri, May 4, 2018 at 8:11 AM, Jamon Camisso via talk <talk@gtalug.org <mailto:talk@gtalug.org>> wrote:
On 2018-05-03 10:44 AM, Alex Beamish via talk wrote: > I'm developing scripts that get run by crontab, so I'm in there making > updates fairly regularly. I would love to be able to document the changes, > so I'm wondering if there a usual and customary technique to version > crontabs?
I've used RCS for lots of ad-hoc systems file tracking, and at least once replaced crontab with a script that added a check-in after calling the real crontab -e. It could easily do a git commit && git push instead. --dave
What are you changing in crontabs so much that you need versioning?
I'm developing EDI scripts, and as development progresses, some of the older scripts are being replaced by newer scripts. On occasion, there are times when I need to disable a crontab job while I solve an issue.
Having version control would be a great back-stop, and would also be useful from a historical point of view. (Why did I disable that job last Thursday? Where did the such and such script go?)
Shouldn't all the logic be in your scripts? Maybe I'm missing something, but it seems like needless complication to me.
What to do and how to do it is in the scripts. How often to do it is in the crontab.
> Ideally there would be some sort of hook around 'crontab -e', but failing > that, I'd have the output of 'crontab -l' (run regularly by cron?) go to a > versioned file. Plan B sounds a bit hokey to me. If you do need to version things, you can just chuck a git repo under /var/spool/cron/crontabs and ignore any files there you don't want tracked.
Under my system, that directory is owned by root, and I prefer not to version anything as root; the code I'm developing is application code, not system code.
The solution I'm leaning towards is to have a shell script for updating the crontab that would do something like 1. pushd $local_crontab_directory 2. crontab -e 3. crontab -l >crontab.$username 4. git diff crontab.$username
If there were changes revealed by the 'git diff', and if I felt they were significant, I could do a git commit, followed by a popd.
Thanks for your questions -- they helped me clarify my understanding of the problem and a possible solution.
-- Alex Beamish
Software Developer / https://ca.linkedin.com/in/alex-beamish-5111ba3 Speaker Wrangler, Toronto Perlmongers / http://to.pm.org/ Baritone, Board Member, Toronto Northern Lights, 2013 Champions / www.northernlightschorus.com <http://www.northernlightschorus.com> Certified Contest Administrator, Barbershop Harmony Society / www.barbershop.org <http://www.barbershop.org>
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
-- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain
On Fri, May 4, 2018 at 9:10 AM, Alex Beamish via talk <talk@gtalug.org> wrote:
On Fri, May 4, 2018 at 8:11 AM, Jamon Camisso via talk <talk@gtalug.org> wrote:
On 2018-05-03 10:44 AM, Alex Beamish via talk wrote:
I'm developing scripts that get run by crontab, so I'm in there making updates fairly regularly. I would love to be able to document the changes, so I'm wondering if there a usual and customary technique to version crontabs?
What are you changing in crontabs so much that you need versioning?
I'm developing EDI scripts, and as development progresses, some of the older scripts are being replaced by newer scripts. On occasion, there are times when I need to disable a crontab job while I solve an issue.
Having version control would be a great back-stop, and would also be useful from a historical point of view. (Why did I disable that job last Thursday? Where did the such and such script go?)
Shouldn't all the logic be in your scripts? Maybe I'm missing something, but it seems like needless complication to me.
What to do and how to do it is in the scripts. How often to do it is in the crontab.
Ideally there would be some sort of hook around 'crontab -e', but failing that, I'd have the output of 'crontab -l' (run regularly by cron?) go to a versioned file. Plan B sounds a bit hokey to me. If you do need to version things, you can just chuck a git repo under /var/spool/cron/crontabs and ignore any files there you don't want tracked.
Under my system, that directory is owned by root, and I prefer not to version anything as root; the code I'm developing is application code, not system code.
I'll share what I shared above again and explain how you don't have to version anything as root with that scheme. Here are the contents of roots/cronjobs/init.sls. cron: service.running: - name: cron - enable: True root-crontab: file.managed: - name: /root/root-crontab - source: salt://cronjobs/root-crontab - template: jinja cmd.run: - name: cd /root; /usr/bin/crontab -r; /usr/bin/crontab root-crontab The "source: salt://cronjobs/root-crontab" line represents where the file "root-crontab" is stored. Let me back up. I have Git repo called "ss" which is cloned/updated into a non-privileged directory. cilkay@jupiter:~/ss$ ls -al total 32 drwxrwxr-x 6 cilkay cilkay 4096 Mar 29 21:12 . drwxrwxr-x. 13 cilkay cilkay 4096 Mar 21 13:15 .. drwxrwxr-x 8 cilkay cilkay 4096 May 4 14:45 .git -rw-rw-r-- 1 cilkay cilkay 203 Mar 29 21:12 .gitignore -rw-rw-r-- 1 cilkay cilkay 303 Mar 29 21:12 minion drwxrwxr-x 2 cilkay cilkay 4096 Mar 21 12:50 minion.d drwxrwxr-x 3 cilkay cilkay 4096 Mar 21 12:50 pillar drwxrwxr-x 19 cilkay cilkay 4096 May 3 16:21 roots In the roots directory, the relevant bit is this. cilkay@jupiter:~/ss$ ls -al roots/cronjobs/ total 16 drwxrwxr-x 2 cilkay cilkay 4096 Mar 21 12:50 . drwxrwxr-x 19 cilkay cilkay 4096 May 3 16:21 .. -rw-rw-r-- 1 cilkay cilkay 303 Mar 29 21:12 init.sls -rw-rw-r-- 1 cilkay cilkay 331 Mar 29 21:12 root-crontab init.sls is above. The file root-crontab contains this. {% set current_env = grains['current_env'] %} PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # Update the server on every reboot @reboot /usr/local/bin/update_tcm.sh {% if current_env == 'dev' or current_env == 'qa' %} # Update the server every day at 0410 10 4 * * * /usr/local/bin/update_tcm.sh {% endif %} The shell script update_tcm.sh contains this. #! /bin/bash # Have to invoke the SaltStack state (ss.init) twice in case # the salt repo has been updated. If it is only updated once # in highstate, it cannot execute any commands or set states # that have changed since highstate was invoked. /usr/bin/python /usr/bin/salt-call --local state.sls ss.init # Must do this to have any custom modules be available. /usr/bin/python /usr/bin/salt-call saltutil.sync_modules /usr/bin/python /usr/bin/salt-call --local state.highstate Salt is in a masterless minion configuration. In other words, this is a self-configuring machine. It does not rely on some external controller. The requirements on this particular project precluded the more conventional master/minion configuration. All the virtual machines are behind corporate firewalls. On first boot or reboot, update_tcm.sh is invoked and Salt runs through all the states to bring the machine to the desired state. That state file referenced in update_tcm.sh, ss.init, bootstraps the whole thing. It does a git update on a specific version. If there are any changes to the root-crontab file, it will be applied by the cronjobs.init state. cronjobs.init maps to ~/ss/roots/cronjobs/init.sls. I could even leave out the ".init" part in cronjobs.init because it's implicit that "init.sls" would be invoked in a state anyway. I like being explicit. cronjobs.init does a few things. in the "cron" section on top, it ensures that the cron daemon is enabled and running. In the root-crontab section, it copies ~/ss/roots/cronjobs/root-crontab to /root/root-crontab, removes the existing crontab for root, and applies whatever is in /root/root-crontab to root's crontab. This happens every time the machine is rebooted or in the case of dev or qa configurations, at 0410 UTC. Of course it can also be done on-demand. Even for a relatively simple requirement like yours, using this approach is perfect because you can start simple and add states incrementally. Eventually, you'll be able to spin up a new server with very little human intervention required. The whole point of this is to have your infrastructure defined as code and kept under revision control. Regards, Clifford Ilkay +1 647-778-8696
participants (8)
-
Alex Beamish -
Christopher Browne -
Clifford Ilkay -
David Collier-Brown -
Jamon Camisso -
Jason Shaw -
Kevin Cozens -
William Park