Skedulord is an attempt to add useful features to cron without reinventing the wheel. It adds a retry mechanic, structured logging and helpful tools to find broken jobs.


Here's what it does.

lord init lord run pyjob "python goodjob.py" - id: c67d713a
- name: pyjob
- tries: 1
- command: python goodjob.py
- startime: 2019-10-13 21:11:03
- endtime: 2019-10-13 21:11:08
- succeed: True
- log: ~/.skedulord/jobs/pyjob/2019-10-13T21:11:03.txt
lord run pyjob "python badjob.py" - id: c67d713a
- name: pyjob
- tries: 3
- command: python badjob.py
- startime: 2019-10-13 21:11:11
- endtime: 2019-10-13 21:11:15
- succeed: False
- log: ~/.skedulord/jobs/pyjob/2019-10-13T21:11:11.txt
tree ~/.skedulord ├── heartbeat.jsonl
└── jobs
    └── pyjob
        ├── 2019-10-13T21:11:03.txt
        └── 2019-10-13T21:11:11.txt
lord serve --host 0.0.0.0 --port 8080 Serving logs over at 0.0.0.0:8080.

This starts a small website with a dashboard of all the jobs. It is easy to find the failed job from here and inspect the logs.

The goal of this project is to offer a lightweight method of organising log files for jobs that run from the command line. It offers a command line app that can be used from cron to help with maintaining an overview.

What does skedulord do?

When you run this command a few things happen.

lord run job1 "python goodjob.py"
  1. It will run `python goodjob.py` and it will assume that `job1` is the name that describes the job.
  2. It will then capture all the stdout that the job produces.
  3. This will be saved at ~/.skedulord/[jobname]/[timestamp].txt.
  4. Then the job metadata is added to ~/.skedulord/heartbeats.jsonl.

When logfiles are nicely stored it becomes easy to view them. Skedulord offers tools to quickly find jobs that failed and locate their logfiles. When combined with cron you suddenly get a lot of features that makes running jobs less scary.

There's some extra settings you can add to a job too.

lord run job1 "python goodjob.py" --retry 3 --wait 5 lord run job2 "python badjob.py" --retry 5 --wait 60

These jobs will be retried should they fail. The first job will attempt 3 tries in total and will wait 5 seconds between each attempt. The second job will attempt 5 tries and wait 60 seconds between each go.

Skedulord commands.

There's some extra helpers in there too.

lord --help Usage: lord [OPTIONS] COMMAND [ARGS]...

SKEDULORD: keeps track of logs so you don't have to.

Options:
--help Show this message and exit.

Commands:
  history  show historical log overview
  init     setup skedulord
  nuke     hard reset of disk state
  run      run (and log) a command, can retry
  serve    start the simple dashboard
  summary  shows a summary of the logs
  version  confirm the version

In particular the `history` command can really help find logs.

lord history +------+-------+----------------------------+
| stat | name  |          logfile           |
+------+-------+----------------------------+
|  x   | pyjob | ../2019-10-18T14:44:30.txt |
|  x   | pyjob | ../2019-10-18T14:44:30.txt |
+------+-------+----------------------------+

There's many settings that can help find the logs you need.

lord history --help Usage: lord history [OPTIONS]

  show historical log overview

Options:
  --failures          only show the failures
  -r, --rows INTEGER  maximum number of rows to show
  -d, --date TEXT     only show specific date
  -j, --jobname TEXT  only show specific jobname

How do I install it?

pip install skedulord

What does skedulord not do?

Skedulord does not:

  1. schedule jobs for you, this is what cron is for
  2. integrate with your compute services on your behalf
  3. write your jobs for you

What are the main usecases?

Lot's of ETL jobs are plain and simple; you just want to make it easy to find logs of jobs that failed. Skedulord adds retry mechanics as well as an overview. Once your jobs become highly dependant then you'll need something more elaborate (like airflow). There's a barrier to entry and maintenance for those tools though so if you don't need it you might prefer this simpler tool.

Food for Thought

How you run your ETL jobs depends on the problems that you have. But there are a few things you might like to consider.


Tips on Cron

Cron documentation can be a bit annoying to find so we've made a small guide here. Note that some of the things we mention here depend on your flavor of linux. So your milage might vary. That said: you can edit the cron on a VM by typing the command `crontab -e`. This will then allow you to write a file in the terminal. There's an example of such a file below:

crontab -e SHELL=/bin/sh
PATH=/home/unixuser/.local/bin
* * * * * lord run test "date >> /tmp/foobar.txt"
* * * * * lord run job "/usr/bin/python3 /path/job.py"

There's a few things that are important to note:

There's an alternative though.

crontab -e SHELL=/bin/bash
* * * * * /path/to/cron_job.sh

cat /path/to/cron_job.sh source /path/to/.bash_profile
lord run download-job "python /path/to/job.py"

This might be a nicer idea since it offers you a bit more of an overview. The bash shell also contains the logic for dependencies which could also contain environment variables and other settings like those that are in .bash_profile. You can also add all sorts of other logic that might connect to extenal sources or cloud compute.


On a Raspberry?

If you can't figure out why cron is not running the right way. Consider installing postfix and also make sure that you're running everything from python 3.

pip3 install skedulord sudo apt-get install postfix

Once this is installed you can set postfix up for local mail. Once this is done you can find error messages from cron.

cat /var/mail/pi

My typical setup is something shown below.

crontab -e 30 * * * * /home/pi/.local/lord run jobname1 "/usr/bin/python3 /path/to/job1.py"
*/4 * * * * /home/pi/.local/lord run jobname2 "/usr/bin/python3 /path/to/job2.py"

Make with love by koaning.io. Github is here.

Docs made with termynal.js.