Trying Out Nomad and Waypoint

bwang -

When deploying onto kubernetes, sometimes it could be hard to totally decouple application management (developer managing apps and scaling) and cluster management (underlying vm, resources), this post hopefully give some structure to my testing of some ways to solve this issue. Edit: On reread of this post, it seems like it's just an excuse for me to try out nomad and waypoint so I have renamed my post title as such.

Nomad

Nomad solves this problem by acting as an online server/system to manage applications on that infra. (Alternative: k8s-terraform) Also, according to the intro talk, it's a really good platform for serverless compute.

What nomad does better than kubernetes:

  • true serverless: maximizes utilization, minimizes resources needed
  • can use windows/legacy systems, nomad isn't limited to kubernetes
  • light: single binary

We can run a simple local server/client with nomad agent -dev -config config.hcl, but there's no way you can do your sithlord "Power! Unlimited Power!" impression with just local compute. The ecosystem might not be as big as google-kubernetes's, but I stole these terraform files, and it looks like I'm using the entire hashicorp ecosystem. It's quite resource intensive so I also scaled down a lot of the infra.

Clients are basically worker nodes, and servers makes sure there's high availability (everything is running across regions); our servers and clients create a cluster of virtual machines that have instances of consul and nomad, etc. We can port-forward (tunnel-through-iap) virtual machine ports to local ports (akin to kubectl port-forward).

I'm connected to my nomad cluster locally and I can just schedule a docker image simply with nomad comands: run/plan/stop, etc. It's analogous with kubectl apply -f, except instead of a yaml, we're submitting a .nomad file.

Nice! I scheduled a hello-world image. I see that it's running in hashistack-client-1. I see that consul automatically picked this guy up in services; the service mesh is all mapped out, etc.

Kubernetes is probably still better in most cases. Even from the case studies (Cloudflare, Other), kubernetes is definitely winning out; it's just easier to find support online and the ecosystem for kubernetes (helm, eks, crds, etc) is just Yeeeeet.

In terms of other kubernetes competitors, there's also rancher (enterprise), docker swarm (lacking some features), mesos (big companies use for fatter infra), etc.

Fargate

I don't have an aws developer account right now because I already trashed two of them so I can't really test this. Fargate is nice because you don't have to manage a cluster. You can deploy containers for compute but apparently it costs more and has more startup overhead.

Waypoint

Hashicorp just announced waypoint; it seems like waypoint is trying to replace the makefile, describing how to build and deploy. First install a waypoint server onto kubernetes, and then run waypoint up to deploy the waypoint service. I wanted to try it out because I can totally see it becoming widely used.

Run waypoint init to add services defined by the waypoint hcl file which you can use to define build, deploy, and release stanzas.

Note that it's perfectly acceptable to do kubectl apply to deploy as seen below. In the waypoint UI, you can also do things like look at the things defined in your hcl file (deploy versions, environments, logs). Waypoint is not yet a mature tool--though I have the tutorial working, the waypoint service has gotten stuck more than once and I'm not sure why (maybe I need to upgrade my cluster?).

  deploy {
    use "exec" {
      command = ["kubectl", "apply", "-f", "<TPL>"]
      template {
        path = "./example-nodejs-exec.yml"
      }
    }
  }

Historically, a developer would have to invest a lot into local setup / deployment makefile, but now you can just run waypoint init/build/deploy/release and run locally, or you can target kubernetes to deploy. You can also add these waypoint commands to build/deploy/release in imperative CI scripts. It's super flexible and it seems like a great developer experience. Reasons not to use waypoint: (1) waypoint hasn't matured yet, (2) it "doesn't make anything better or faster"; if existing tooling does the job, I think the argument for adapting waypoint is weakened.

Down the line, it seems like they want people to use consul in tandem with waypoint for the full life-cycle treatment of a service, being able to target development and production releases.

Other things:

  • people need to use google cloud shell or other cloud terminal equivalents more when writing tutorials like they do here since everything is gonna start running on cloud
  • api if you just want nomad to be some compute
  • Nomad Links: Good list of tooling, example with binary instead of docker, a nice templating walkthrough, a library of reference nomad files
  • it's kinda romantic how they describe parts of config files as stanzas