How To Create Cron Jobs In Python

In this post I am going to explain couple of different approaches in creating cron jobs in python

If you are reading this article you know what is a cron job and what it does. In this article I want to focus on how you can create a cron job (or schedule a task basically) programmatically through python. This would be useful in many cases, for instance if you need to schedule tasks based on user requests or send them a reminder on a monthly basis.

There are two different approaches to this, you can either host your own scheduler or you can use a third-party service for it. If you want to host your own scheduler you can use the following python libraries:

Schedule

Schedule is a third-party library for python that can be used to schedule tasks. To install this library you can run:

pip install schedule
              

and then you can use it like this:

import schedule
import time

def task():
   print("Running task now...")

schedule.every(10).seconds.do(task)

while True:
   schedule.run_pending()
   time.sleep(1)
               

If you like to use python decorator you can code the above example like this:

import time
from schedule import every, repeat, run_pending

@repeat(every(10).seconds)
def task():
   print("Running task now...")

while True:
   run_pending()
   time.sleep(1)
               

Note that you need to keep the program up and running, that's why we have the while loop. You can read more about the schedule library here

APScheduler

APScheduler is another third-party library that is similar to scheduler. But it has more features and backend compatibility for production environment, in my opinion. To install you can run:

pip install apscheduler
               

And this is one example on how you can use it to schedule a task:

from datetime import datetime
import time
import os
from apscheduler.schedulers.background import BackgroundScheduler


def task():
   print('Running task now...')



scheduler = BackgroundScheduler()
scheduler.add_job(task, 'interval', seconds=10)
scheduler.start()

try:
   while True:
       time.sleep(1)
except Exception:
   scheduler.shutdown()
              

You can read mode about this library here.

python-crontab

python-crontab is another third-party library that you can use to schedule jobs. This one is a direct API that lets you access the system cron directly and add/remove cron jobs to it. To install this library, you can run:

pip install python-crontab
               

and then you can use it like this:

from crontab import CronTab
cron = CronTab(user='root')
job = cron.new(command='echo this_is_from_cron')
job.minute.every(10)
cron.write()
              

In this case if you want to run a specific python code at the scheduled time, you need to have it in a separate file and pass the path to python, like this:

 job = cron.new(command='/usr/bin/python /absolute/path/to/script.py')
               

Scheduling cronjob on system crontab can have some advantages, for instance, you won't need to maintain a process up and running all the time like previous examples. However, it may not be suitable for all cases since you have to run your python scripts as separate codes. Specially if you have to pass different arguments to each task you are scheduling.

If you don't want to maintain your own scheduler and you don't want to use system cron either, you can use one of the following cloud services:

Cron Job on AWS

If you are already using AWS this could be an option for you, however it could be hard to maintain and overly complicated for many use cases. The idea here is that you can schedule "Rule"s in AWS that run "Lambda function"s. So in this case, what you need to do first is to create an AWS lambda function that contains the piece of code you want to run periodically. Then in your code you can have something like this to schedule a job that runs that lambda function:

import boto3

events_client = boto3.client('events')
lambda_fn_arn = "your-lambda-function-arn"
cron = "*/10 * * * *"
rule_name = "my_rule"

rule_res = events_client.put_rule(
  Name=rule_name,
  ScheduleExpression='cron({})'.format(cron),
  State='ENABLED',
)
events_client.put_targets(
  Rule=rule_name,
  Targets=[
    {
      'Id': "some_id",
      'Arn': lambda_fn_arn,
    },
  ]
)
               

The above code basically creates a new "Rule" with a "ScheduleExpression" on AWS and then creates a "Target" that connects that "Rule" to the lambda function you have created.

CronDock

CronDock is my "container native cloud based cron and workflow" service. It is a serverless tool for running containerized code on demand or based on a schedule. In this case, you are going to need a docker hub account which you use to host your containerized code. I also recommend you to install docker cli which makes it easier to update and push new version of your container to docker hub.

Now let's say you want to run the following code on daily basis:

import requests
import sys

result = requests.get(sys.argv[1])
sys.stdout.write(result.text)
               

And let's say you have this code in a file called "example.py". To containerize this code using docker, you need to add a file called "Dockerfile" in the same directory and add the following commands to it:

FROM python:3
RUN pip install requests
COPY example.py ./
ENTRYPOINT ["python" ,"example.py"]
              

This basically says to docker to create a container image that has python 3 installed on it, then "pip install" the "requests" library on the image and then copy the "example.py" file to the container. Finally, run "python example.py" when this image is called.

Now to containerize this code using docker cli you first need to login to your docker hub account by running the following command (you need to replace yourusername with your docker hub username and yourpassword with your docker hub password):

docker login --username yourusername --password yourpassword
               

Then run the following command to containerize the code (don't forget to replace yourusername):

docker build -t yourusername/example:latest -f Dockerfile .
               

This command basically creates a containerized image of the code based on the instruction provided in "Dockerfile". If you want to run the image locally to test, you can run the following command (don't forget to replace yourusername):

docker run --rm yourusername/example https://crondock.com/
               

Finally, to push this image to docker hub you can to run the following command (again don't forget to replace yourusername):

docker push yourusername/example
               

Now the containerized image of the code lives on docker hub and CronDock can have access to it. The next step is creating a cron job to run this image on daily basis. For that, you are going to need a CronDock account, which you can create here

After signing up, CronDock will provide you with an "API key" which you will use to access CronDock API. Now you can create a cron job by simply calling CronDock API using your API key. In python, you can create a daily cron job for the above image like this (you have to replace XXXXX with your API key and yourusername with your docker hub username):

headers = {
    "Content-Type": "application/json",
    "Authorization": "Api-Key XXXXX"
}

data = {
    "name": "example-cron",
    "image": "yourusername/example",
    "type": "cron",
    "args": ["https://crondock.com/"],
	"schedule": "0 0 * * *"
  }

result = requests.post('https://api.crondock.com/task/', headers=headers, json=data)
print(result.text)
              

And running this code you should see an output like this:

{
	"status": "OK",
	"data": {
		"id": 22,
		"created_at": "2022-08-01T15:54:45.316811Z",
		"name": "example-cron-wmnrnf",
		"image": "yourusername/example",
		"args": ["https://crondock.com/"],
		"type": "cron",
		"schedule": "0 0 * * *",
		"user": 2,
		"workflow_task": null,
		"workflow_result": null,
		"secret": null
	}
}
              

In this code we are using "requests" library to submit a "post" request to CronDock API end point "https://api.crondock.com/" to create a "task" with type set as "cron". In the task, we want to run the "yourusername/example" image and pass the "https://crondock.com/" as the input argument to the image. And we are setting the "schedule" to every day at 00:00 UTC time (you can use this tool for cron format schedules). Now, every day at 00:00 UTC, CronDock will pull the "yourusername/example" image from docker hub and runs the image passing "https://crondock.com/" as an argument to it.

In some cases, you need to run multiple images with some dependencies, for instance you may have to run a specific code in case the first one fails. In such cases you can use CronDock workflow which I explained in more details here: https://crondock.com/how-to-create-a-workflow-programmatically.html

Please let me know if you have any question at support@crondock.com

I also have a good document which provides more detailed information on all the requests you can make to CronDock API.