Continuous Delivery with Containers Nick Gauthier @ngauthier Developer @codeship

What is Continuous Delivery?

Continuous Delivery (CD) is a software engineering approach in which teams keep producing valuable software in short cycles and ensure that the software can be reliably released at any time. Chen, Lianping (2015). "Continuous Delivery: Huge Benefits, but Challenges Too". IEEE Software 32

“valuable software” I think we can all agree here!

“short cycles” ship early, ship often!

“reliably released at any time” don’t break the build!

What are our needs?

Core CD System Attributes 1. 2. 3. 4.

Speed Parity Reliability Flexibility

Speed: The CD Cycle

1. 2. 3. 4. 5.

Check out code Run tests Build deployment artifact Ship deployment artifact Start deployment artifact

Deployment Artifact? 1. 2. 3. 4. 5. 6. 7. 8.

Binary “slug” WAR VM AMI Container tar etc...

Parity “it worked for me”

Reliability “try running it again…”

Flexibility “I need X”

What are our CD options?

PaaS

1. 2. 3. 4.

Speed ✔ Parity ✘ Reliability ✔ Flexibility ✘

Config Management (puppet, chef, ansible, salt)

1. 2. 3. 4.

Speed ✘ Parity ✘ (✔ if virtualized locally) Reliability ✔? Flexibility ✔

VM

1. 2. 3. 4.

Speed ✘ Parity ✔ Reliability ✔ Flexibility ✔

Container

1. 2. 3. 4.

Speed ✔ Parity ✔ Reliability ✔ Flexibility ✔

Docker

1. 2. 3. 4.

Speed ✔✔ Parity ✔ Reliability ✔ Flexibility ✔

Why Docker?

Cache layers rebuilds are fast!

Dockerfile simple, understandable, centralized

Official + Community Images don’t reinvent the wheel unless you want to

Hosting Choices low lock-in, many new options including OSS diy

Docker CD What are our options?

#1 Local

LOCAL

1. 2. 3. 4.

Run tests Build images Push images Trigger host pull

LOCAL

1. 2. 3. 4.

Run tests (hopefully? same env?) Build images (slow, cpu + net) Push images (slow, race?) Trigger host pull (race?)

#2 Hosted Solution

HOSTED SOLUTION

1. 2. 3. 4. 5.

Build CI Container Run tests in CI Container Build Production Container Push Production Container Trigger host pull

HOSTED SOLUTION

1. 2. 3. 4. 5.

Build CI Container (caching? multiple?) Run tests in CI Container (prod parity?) Build Production Container (build artifacts? mult?) Push Production Container Trigger host pull

#3 Custom Solution

Custom Solution ???

What are we missing?

Centralized CD Our goals

Parity

PARITY

CI Container == Prod Container

PARITY

CI Container ~= Prod Container

PARITY

Same FROM

PARITY

Same Packages (+ a few)

PARITY

Linked Services (not monoliths)

PARITY

Hosting “Stack” = Docker

Orchestration

ORCHESTRATION

Running tasks + linking services

ORCHESTRATION

Docker compose run

ORCHESTRATION

Multiple Compositions? (unit + integration + acceptance)

Speed

Speed (caching)

SPEED + CACHING

Base Images

SPEED + CACHING

Layers from “last time”

SPEED + CACHING

Layers from “last time” = registries!

Parallelism

Parallelism @#$%!

PARALLELISM

Docker isolation!

PARALLELISM

Orchestration

PARALLELISM

Orchestration of Orchestration

PARALLELISM

Pipelines Checkout Static Analysis Lint

Style

Security

CI Build A U1

U2

U3

Build B U4

I1

I1

Build C I1

A1

CD Build P1

Build P2

Push P1

Push P2

Deploy P1

Deploy P2 PASS!

A1

PARALLELISM

Pipelines Compose Checkout Static Analysis DB Lint

Cache Style

Security

CI Build A U1

U2

U3 API

Build B U4

I1

I1 I1 APP + Tests

Build C A1

CD Build P1

Build P2

Push P1

Push P2

Deploy P1

Deploy P2 PASS!

A1

Let’s build it! a.k.a. lessons learned a.k.a. what went wrong :)

Step 1: Define Containers

Official Language Containers

Official Language Containers if you can

Official Language Containers dictate O.S. (often debian)

Official Language Containers big, pull = slow, lots of disk (esp w/ docker machine + vbox)

Containers for CI

Containers for CI code changes frequently

Containers for CI code changes frequently = lots of rebuilds

Containers for CI dependencies = majority of time

Containers for CI dependencies = change occasionally

Hack #1: Layer Dependencies

# Layered ADD Godeps + Makefile / Gemfile + .lock / … RUN make deps / bundle / … ADD . CMD [“run my tests”]

CODE SNIPPET

# Classic ADD . RUN make deps / bundle / … CMD [“run my tests”]

Dependency Layer Cached on “manifest” files

Dependency Layer Great on CI Fantastic on dev machine

Hack #2: CI Registries

Pull app:branch before CI

Pull app:branch before CI Primes all layers from last time

Push app:branch after CI Updates layers in registry

Run registry near CI You can use docker hub but you don’t have to

Production Containers Same as CI (-pkgs)

Production Containers Layering + Common bases help here too!

Production Containers Huge = slow push = slow host pull = slow deploy

Hack #3: Static Containers

# Dockerfile.production FROM debian:latest RUN apt-get update && \ apt-get install some-dyn-lib ADD prod-binary / CMD [“/prod-binary”]

CODE SNIPPET

# Commands during CI make prod-binary docker build -f Dockerfile.production .

Static Container

1. Minimal Packages 2. No language packages 3. Only works for compiled languages (but also has gains for vm languages)

Hack #3.a: Scratch Containers

Scratch? empty tarball

# Dockerfile.production FROM scratch ADD prod-binary / CMD [“/prod-binary”]

CODE SNIPPET

# Commands during CI make prod-binary docker build -f Dockerfile.production .

Hilariously Tiny single digits mb

Scratch Containers

1. Only works for statically compiled binaries 2. Worst CI + Prod parity 3. Near Instant push+pull+boot+deploy

Minimal Guests

Minimal Guest

1. 2. 3. 4.

Bare O.S. Minimal toolchain Minimal Packaging + DIY Small containers for the dynamic crowd

Promising Project gliderlabs/docker-alpine

CODE SNIPPET

# 16mb mysql container FROM gliderlabs/alpine:3.1 RUN apk --update add mysql-client ENTRYPOINT ["mysql"]

alpine is just one hopefully we will have many options!

Example: codeship-ruby https://github.com/codeship-library/ruby 718mb -> 349mb

Orchestration

docker compose

# running ci docker compose run app make test

CODE SNIPPET

# docker-compose.yml app: build: . links: - db ports: - "8000:8000" db: image: postgres

docker compose problems Can’t tell when services are running

docker compose problems ENV encryption

docker compose problems only 1 at a time

Docker CI we had to manually run and link :(

CODE SNIPPET

docker run -d --name u1-postgres postgres sleep magicnumber # :( docker run --name u1-app --link u1-postgres:postgres \ app make test docker kill u1-postgres

Parallelism

Parallelism Manually run and link and namespace

Parallelism run your own agent split work monitor containers

Parallelism @#$%!

Parallelism we’ll figure it out!

Docker Swarm

Docker Swarm if all we use to run CI is Docker, Swarm gives us machine parallelism

Deploy

Deployment Styles

1. Push code + build on host (e.g. Deis, EB, …) 2. Build + push image + trigger host pull (e.g. EB, ECS, GAE, …) 3. Build + crossbuild for host + push (e.g. Heroku docker release)

Build on Host

1. 2. 3. 4.

Removes burden from CI Must be single-Dockerfile app Must be deployable in isolation Can’t leverage local cache

Build + Push + Trigger Pull

1. 2. 3. 4.

Use local cache CI must do production builds Push + Pull adds time Can push many and launch many

Build + Crossbuild

1. ???

Hack #4: use the same registry

Use the same registry

1. 2. 3. 4. 5.

Put registry near CI Push CI layers to registry a la Hack #2 Minimal layers pushed with small changes Put registry near host Minimal pull onto host with small changes

Keep common bases hot

Try to use similar bases

CI + CD + Hosting keep them close

CI + CD + Hosting keep them close (a.k.a. Amazon all the things)

What’s next?

Better parallel tooling

Better development environments (non virtualized drivers)

Better machine provisioning (docker machine as a library)

Codeship Docker pages.codeship.com/docker (or come say hi!)

Thanks!

Questions?