Today we review Earthly (https://earthly.dev), a build system designed for modern container-centric workflows.

It strives for ease of use and to be familiar to developers coming from traditional build tools like Make and/or from container-based workflows using Docker.

It is also designed to supports the plethora of build tools native to each ecosystem (cmake, npm/yarn, python setup.py, go build, cargo) instead of trying to replace them.

And, as we’ll see, it provides compelling value when integrated into existing Continuous Integration / Continuous Delivery (CI/CD) systems.

Inspired by Make

The first and most striking aspect of Earthly is that it looks like a hybrid between Make and Docker.

Let’s see a sample Earthfile, the file format in which builds are described :

VERSION 0.7
FROM golang:1.21
WORKDIR /go-workdir

deps:
    COPY go.mod go.sum ./
    RUN go mod download
    # Output these back in case go mod download changes them.
    SAVE ARTIFACT go.mod AS LOCAL go.mod
    SAVE ARTIFACT go.sum AS LOCAL go.sum

build:
    FROM +deps
    COPY --dir cmd/ .
    RUN go build -v -o $GOPATH/bin/device-app cmd/main.go
    SAVE ARTIFACT $GOPATH/bin/device-app

docker:
    FROM alpine:3.14
    ARG tag='latest'
    COPY +build/device-app /go/bin/
    ENTRYPOINT ["/go/bin/device-app"]
    SAVE IMAGE device-app:$tag

The first section declares the version of the Earthly syntax used in the file,
followed by the base Docker container image used for the build:

FROM golang:1.21

Then the file describes three targets that handle different aspects of the build:

deps downloads all project dependencies described in go.mod and go.sum
build compiles main.go into an executable
and docker packages the executable as a container image

And each target, in turn, is composed of a list of commands (also known as a recipe) that are executed in order.
For example:

build:
    FROM +deps
    COPY --dir cmd/ .
    RUN go build -v -o $GOPATH/bin/device-app cmd/main.go
    SAVE ARTIFACT $GOPATH/bin/device-app

With a Docker twist

Now, the list of commands we just saw look suspiciously similar to Dockerfile commands.
For example:

    FROM +deps
    COPY --dir cmd/ .
    RUN go build -v -o $GOPATH/bin/device-app cmd/main.go

Familiar commands

This was a deliberate choice by the Earthly team, to make the tool familiar to developers coming from Docker-based workflows.

As of October 2023, Earthly supports 34 commands, and the full list is available at https://docs.earthly.dev/docs/earthfile.

The most common commands allow to:

Start from a base image (FROM) or the contents of a previously-built target (FROM +deps)
Copy files from the host to the container (COPY), from one stage to another and even from builds described in an external repository
Run shell commands (RUN)
Trigger other builds (BUILD +deps)
Save build artifacts (SAVE ARTIFACT) and images (SAVE IMAGE)
and more…

With a caveat

While commands have been designed to be familiar to developers who have worked with Docker containers, they are not exactly Dockerfile commands.

For example COPY works slightly differently and requires a --dir option for copying multiple directories.

So, while the learning curve is not steep, developers need to be aware of the differences.

Developer experience

Earthly provides a number of features that significantly improve the developer experience.

Local and CI/CD friendly

Earthly is designed to be used locally, on the developer’s machine, and to be integrated with CI/CD systems.

On a local machine, the developer can invoke a target with the earthly command:

$ earthly +build

Once builds and tests are verified locally, the same command can be used in CI/CD systems, which simplifies integration.

Tracking of dependencies

The tool tracks dependencies in a different way than Make.

It infers dependencies from the commands themselves, instead of relying on the developer to explicitly list them.

For example, the build target in the Earthfile above

depends on the deps target because it uses the FROM +deps command and
it depends on all files in cmd/, including cmd/main.go because it uses the COPY --dir cmd/ . command.

This is quite different from Makefiles, where dependencies (a.k.a. prerequisites) are listed explicitly on the right of each target.
For example:

$GOPATH/bin/device-app: cmd/main.go
    go build -v -o $GOPATH/bin/device-app cmd/main.go

means that a $GOPATH/bin/device-app target has cmd/main.go as a prerequisite / dependency.
If the prerequisite is newer than the target itself, the target will be rebuilt by executing go build -v -o $GOPATH/bin/device-app cmd/main.go

And note that in Earthfiles targets are symbolic names while in Makefile targets and prerequisites are interpreted by default as actual file paths. If the developer wants names that are not file paths, they need to declare them as .PHONY targets.

Effective caching

Another interesting aspect of Earthly is that it caches build steps aggressively.

This means that if you change a step, only that step and the steps that depend on it will be rebuilt.

And the best part? Even steps across different Earthfiles that are the same (let’s say installation of the same common build tools on top of the same base image) will be cached.

And parallel builds by default

Targets that don’t depend on each other are executed in parallel.

All the features above, combined, result in faster builds and an excellent developer experience.

But why not just use Make and Docker?

It’s a valid question.
The Earthfile above could be replaced by a multi-stage Dockerfile:

# Build stage
FROM golang:1.21 AS builder
WORKDIR /go-workdir
COPY go.mod go.sum ./
RUN go mod download
COPY cmd/ .
RUN go build -v -o $GOPATH/bin/device-app cmd/main.go

# Runtime image
FROM alpine:3.14
COPY --from=builder /go/bin/device-app /go/bin/
ENTRYPOINT ["/go/bin/device-app"]

and a Makefile:

docker: go.mod go.sum cmd/main.go
    docker build -t device-app .

.PHONY: docker

where the command to build a docker image is make docker, instead of earthly +docker.

Is it equivalent though?
There are a few drawbacks already visible:

The Makefile needs to list all dependencies explicitly, and that’s error prone
As we have seen, Earthly infers dependencies from the commands themselves
Any step listed in Makefiles is not cached. Only steps in Dockerfile take advantage of Docker’s caching mechanism
And a major one for developer experience, intermediate targets like deps are not simple to model in the Dockerfile and Makefile and can’t be invoked directly while developing the build

In conclusion

Earthly is a promising build system that provides a number of features that improve the developer experience and the performance of CI/CD systems.

It is still a young project, so needs to be evaluated carefully before being adopted in production, but it’s worth keeping an eye on it.

Earthly - A fast and effective build system