Skip to main content

· 4 min read

It's that time of the year again, and the 2022 Advent of Code is up and running! If you're unfamiliar with it, the Advent of Code (AoC) is a series of programming puzzles that runs from December 1 - 25; one question a day, with two parts each. You can claim points on a leader board by being the fastest to solve a problem, but I generally just do them for fun.

I don't think I've ever finished all 25 days before, due to inevitably falling too far behind over the course of the month, but I have a fondness for it, because it pushed me to use a very valuable skill, that I try to use every day: solve the next question before it's asked. Or, you know, try to.

When I first started participating, as a casual player, I would usually try to solve the problem with one-liners, and add operating logic while mapping through steps that was as close to a series of one-liners as I could make it. Then, for the second part, I'd find myself doing mostly the same, from scratch, to solve the twist on the original question that the second part asked for. It was a real slog. Eventually, I stopped trying to solve the first question for what was asked, and started to solve it for what I thought the second question was going to be.

That's what I consider the third, and most challenging problem; solving the first one in such a way that the code can be (mostly) re-used in anticipation of the second part with minimal re-write, but also not over-engineering a bloated solution that would be a pain to maintain. Also, it's the most fun problem to solve! It's always exciting to get it right, and worse case you've invested yourself deeper in the problem-space you're trying to solve.

Here's an example of from Day 4. The premise of the question is that some elves are assigned rooms to clean, but it's noticed that, given a pair of elves, some of their assigned rooms overlap. The (literal) question to part one:

In how many assignment pairs does one range fully contain the other?

The straight-forward thing to do, as we iterate over pairs of elves, is to check does one completely overlap, or not?

def redundant: Boolean =
(workers._1.assignedTo.toSet, workers._2.assignedTo.toSet) match {
case (a, b) if a.subsetOf(b) | b.subsetOf(a) => true
case _ => false
}

And that's the right solution! It will get you the answer. Ship it! But, perhaps ask yourself, "What's a next logical question I might be asked once I know if there has been overlapping rooms assigned?". My solution for this part of the question answers if there is a redundancy, and if so, which elf is redundant.

def redundant: Option[Int] =
(workers._1.assignedTo.toSet, workers._2.assignedTo.toSet) match {
case (a, b) if a.subsetOf(b) => Some(1)
case (a, b) if b.subsetOf(a) => Some(2)
case _ => None
}

That solution serves the same logic as the first - I can check if the Option is defined or not for a true/false solution, but now I have the added power of peaking into the result if I need to know which one to re-assign. Spoiler alert, the follow up question didn't ask about it this time.

I've found these little pieces of logic can start to add up in a larger code base encompassing some domain, and when you're asked how long it would take to roll out some new feature, instead of saying "2 weeks", you can say, "We'll, it's funny you should ask that..."

If you're interested in my full solution for day 4, you can find it here.

· One min read

My solution for https://adventofcode.com/2022/day/3

import zio.*
import zio.stream.*

object Day3 extends ZIOAppDefault {

val source: String => ZStream[Any, Throwable, String] =
fileName =>
ZStream
.fromFileName(fileName)
.via(ZPipeline.utfDecode >>> ZPipeline.splitLines)
.filter(_.nonEmpty)

// Indexes start at 0. Avoid an extra map by prepending an element, and taking the tail.
val priorities: Map[Char, Int] =
(('a' +: ('a' to 'z')) ++ ('A' to 'Z')).zipWithIndex.tail.toMap


val data = "day-3-1.data"
override def run: ZIO[Any, Any, Any] = for {
_ <- source(data)
.map(str => str.splitAt(str.length / 2)) // Split in two
.map { case (a, b) => a.intersect(b).head } // Elfs promised there would be only one
.map(k => priorities.getOrElse(k, throw new RuntimeException("hau ruck")))
.run(ZSink.sum)
.debug("Answer pt.1")
_ <- source(data)
.grouped(3) // Get it together, Elves!!!
.map(_.reduceLeft((a, b) => a.intersect(b)).head) // Elfs promised there would be only one
.map(k => priorities.getOrElse(k, throw new RuntimeException("hau ruck")))
.run(ZSink.sum)
.debug("Answer pt.2")
} yield Exit.Success

}

· 3 min read

My solution for https://adventofcode.com/2022/day/2

import zio.*
import zio.stream.*

object Day2 extends ZIOAppDefault {

sealed trait RPS
object RPS {

case object Rock extends RPS
case object Paper extends RPS
case object Scissors extends RPS

def apply(s: String): RPS = s match {
case "A" | "X" => Rock
case "B" | "Y" => Paper
case "C" | "Z" => Scissors
case _ => throw new RuntimeException("That's cheating!")
}

extension(rps: RPS) {
def losesTo: RPS = rps match {
case Rock => Paper
case Paper => Scissors
case Scissors => Rock
}
def winsAgainst = rps match {
case Rock => Scissors
case Paper => Rock
case Scissors => Paper
}
}

def score(a: RPS, b: RPS): Int = {
val choice = b match {
case Rock => 1
case Paper => 2
case Scissors => 3
}
val outcome = (a, b) match {
case (x, y) if x == y => 3 // Draw
case (x, Rock) if x == Scissors => 6 // Winner
case (x, Paper) if x == Rock => 6 // Winner
case (x, Scissors) if x == Paper => 6 // Chicken substitute dinner
case _ => 0 // If ya ain't first, yer last
}
// A refactor of outcome, after extension methods were added for part 2
val betterOutcome = (a, b) match {
case (x, y) if x == y => 3 // Draw
case (x, y) if x.losesTo == y => 6 // Winner
case _ => 0 // If ya ain't first, yer last
}
choice + betterOutcome
}

def throwTheGame(a: RPS, b: RPS): Int = (a, b) match {
case (x, Rock) => score(x, x.winsAgainst) // Need to lose
case (x, Paper) => score(x, x) // Need to tie
case (x, Scissors) => score(x, x.losesTo) // Need to win
}

}

val source: String => ZStream[Any, Throwable, String] =
fileName => ZStream.fromFileName(fileName).via(ZPipeline.utfDecode)

val pt1Pipeline: ZPipeline[Any, Nothing, String, Int] =
ZPipeline.splitLines >>>
ZPipeline.map[String, Int] { str =>
val arr = str.split(" ").map(RPS.apply)
RPS.score(arr.head, arr.last)
}

val pt2Pipeline: ZPipeline[Any, Nothing, String, Int] =
ZPipeline.splitLines >>>
ZPipeline.map[String, Int] { str =>
val arr = str.split(" ").map(RPS.apply)
RPS.throwTheGame(arr.head, arr.last)
}

val scoreSink: ZSink[Any, Nothing, Int, Nothing, Int] = ZSink.sum[Int]

val data = "day-2-1.data"
override def run: ZIO[Any, Any, Any] = for {
_ <- source(data)
.via(pt1Pipeline)
.run(scoreSink)
.debug("Answer pt.1")
_ <- source(data)
.via(pt2Pipeline)
.run(scoreSink)
.debug("Answer pt.2")
} yield ExitCode.success

}

· One min read

My solution for https://adventofcode.com/2022/day/1

import zio.*
import zio.stream.*

object Day1 extends ZIOAppDefault {

// Add a couple new lines so we don't have to collectLeftover
val source: String => ZStream[Any, Throwable, String] =
fileName =>
ZStream.fromFileName(fileName).via(ZPipeline.utfDecode) ++ ZStream("\n\n")

val pt1Pipeline: ZPipeline[Any, Nothing, String, Int] =
// Split by lines
ZPipeline.splitLines >>>
// group into Chunk[Chunk[_]] based on empty lines
ZPipeline
.mapAccum[String, Chunk[String], Chunk[String]] {
Chunk.empty
} { (state, elem) =>
if (elem.nonEmpty) (state :+ elem, Chunk.empty)
else (Chunk.empty, state)
}
//Get rid of empty Chunks
.filter(_.nonEmpty) >>>
// Add up the inner Chunk
ZPipeline.map[Chunk[String], Int](_.map(_.toInt).sum)

// Find the max
val pt1Sink: ZSink[Any, Nothing, Int, Nothing, Int] =
ZSink.collectAll[Int].map(_.max)

// Sum of biggest 3
val pt2Sink: ZSink[Any, Nothing, Int, Nothing, Int] =
ZSink.collectAll[Int].map(_.sortBy(-_).take(3).sum)

override def run: ZIO[Any, Any, Any] = for {
_ <- source("day-1-1.data")
.via(pt1Pipeline)
.run(pt1Sink)
.debug("Answer pt.1")
_ <- source("day-1-1.data")
.via(pt1Pipeline)
.run(pt2Sink)
.debug("Answer pt.2")
} yield ExitCode.success

}

· 8 min read

Hello, class. Today we're going to use sbt to publish artifacts to GitHub packages via GitHub Actions when we tag/release our codebase, and we're not going to use any sbt plugins to do it!

It's not that scary

If you check the official SBT Documentation, you can see that the main things you need to do are specify where you are going to publish

publishTo := Some("Sonatype Snapshots Nexus" at "https://oss.sonatype.org/content/repositories/snapshots")

and how to authenticate with that repository

credentials += Credentials("Sonatype Nexus Repository Manager", "my.artifact.repo.net", "admin", "admin123")

that's it!™️

Lead by example

I'm working on a (very new) project that's a slim framework to build ZIO based CLI apps called ursula, so I will use this as an example, and talk through the build.sbt file, and what the important "gotchas" are. The general plan is:

  1. Show the full build.sbt file
  2. Discuss parsing tags to artifact versions using default environment variables
  3. Configure SBT to publish to our repositories package endpoint
  4. Cover some SBT gotchas

1 The full build.dbt

The general structure of this project is that the main library lives in a project/folder named ursual, and there is an example project that depends on it. We'll cover this in the "gotchas", but there is not a root project.

val tagWithQualifier: String => String => String =
qualifier =>
tagVersion => s"%s.%s.%s-${qualifier}%s".format(tagVersion.split("\\."): _*)

val tagAlpha: String => String = tagWithQualifier("a")
val tagBeta: String => String = tagWithQualifier("b")
val tagMilestone: String => String = tagWithQualifier("m")
val tagRC: String => String = tagWithQualifier("rc")

val defaultVersion: String = "0.0.0-a0"
val versionFromTag: String = sys.env
.get("GITHUB_REF_TYPE")
.filter(_ == "tag")
.flatMap(_ => sys.env.get("GITHUB_REF_NAME"))
.flatMap { t =>
t.headOption.map {
case 'a' => tagAlpha(t.tail) // Alpha build, a1.2.3.4
case 'b' => tagBeta(t.tail) // Beta build, b1.2.3.4
case 'm' => tagMilestone(t.tail) // Milestone build, m1.2.3.4
case 'r' => tagRC(t.tail) // RC build, r1.2.3.4
case 'v' => t.tail // Production build, should be v1.2.3
case _ => defaultVersion
}
}
.getOrElse(defaultVersion)

ThisBuild / organization := "com.alterationx10"
ThisBuild / version := versionFromTag
ThisBuild / scalaVersion := "2.13.8"
ThisBuild / publish / skip := true
ThisBuild / publishMavenStyle := true
ThisBuild / versionScheme := Some("early-semver")
ThisBuild / publishTo := Some(
"GitHub Package Registry " at "https://maven.pkg.github.com/alterationx10/ursula"
)
ThisBuild / credentials += Credentials(
"GitHub Package Registry", // realm
"maven.pkg.github.com", // host
"alterationx10", // user
sys.env.getOrElse("GITHUB_TOKEN", "abc123") // password
)

lazy val ursula = project
.in(file("ursula"))
.settings(
name := "ursula",
libraryDependencies ++= Seq(
"dev.zio" %% "zio" % "2.0.0-RC6"
),
fork := true,
publish / skip := false
)

lazy val example = project
.in(file("example"))
.settings(
publishArtifact := false,
fork := true
)
.dependsOn(ursula)

2 Setting the package version

Note that this section is more about how I am deploying versions for packages. You likely already have a versioning scheme, and are handling that mapping, but here you go anyway 😆

Maven as a version ordering specification that we'll use for non-numeric qualifiers, which has this ordering:

"alpha" < "beta" < "milestone" < "rc" = "cr" < "snapshot" < "" = "final" = "ga" < "sp"

In all honesty, for simple projects this many qualifiers is probably overkill! I've mapped out alpha, beta , milestone, rc and "" (which is no qualifier, or "final"/"ga").

A note about GitHub packages that was true the last time I tried publishing SNAPSHOTS (not sure if this is still the case), but they do not allow you to overwrite a package - so to publish over top of an existing SNAPSHOT - you'd need to delete it first, and upload the new one. That's more work than it's worth, so I've designated alphas as my "snapshots"

With that in mind, I want to use git tags to map to these, so, for example, I've designated that tags a.1.2.3.4 should build with version 1.2.3-a4. So by providing a different initial character (a/b/m/r), I can control what qualifier it's release as.

With that outlined, I can achieve this with the tagWithQualifier function below (and it's helpers).

val tagWithQualifier: String => String => String =
qualifier =>
tagVersion => s"%s.%s.%s-${qualifier}%s".format(tagVersion.split("\\."): _*)

val tagAlpha: String => String = tagWithQualifier("a")
val tagBeta: String => String = tagWithQualifier("b")
val tagMilestone: String => String = tagWithQualifier("m")
val tagRC: String => String = tagWithQualifier("rc")

And when I want to do a "production release", I just use the common v1.2.3 tag.

We will use default environment variables to read the git tags, so we can parse them.

We will check + filter for GITHUB_REF_TYPE; this can be branch or tag (we want tag). If we made it this far, we will then check GITHUB_REF_NAME - which at this point, should be the value of out git tag.

val defaultVersion: String = "0.0.0-a0"
val versionFromTag: String = sys.env
.get("GITHUB_REF_TYPE")
.filter(_ == "tag")
.flatMap(_ => sys.env.get("GITHUB_REF_NAME"))
.flatMap { t =>
t.headOption.map {
case 'a' => tagAlpha(t.tail) // Alpha build, a1.2.3.4
case 'b' => tagBeta(t.tail) // Beta build, b1.2.3.4
case 'm' => tagMilestone(t.tail) // Milestone build, m1.2.3.4
case 'r' => tagRC(t.tail) // RC build, r1.2.3.4
case 'v' => t.tail // Production build, should be v1.2.3
case _ => defaultVersion
}
}
.getOrElse(defaultVersion)

Now we have a way to dynamically set the version published based on git tagging!

ThisBuild / version := versionFromTag

3 Where to publish

We need to set our publishTo and credentials. For the publishTo, GitHub has the structure "https://maven.pkg.github.com/USER/REP", so just update with your information. This pattern should hold for orgs as well. An important thing to note is the realm "GitHub Package Registry". This is handled automatically, but when publishing hits the repository, it'll give back a 401 and tell you how you should authenticate and what the realm is. The significant thing to note, is that the value here for the realm is fixed, and determined by the hosting server. sbt will use this realm to find the matching set of credentials.

ThisBuild / publishTo := Some(
"GitHub Package Registry" at "https://maven.pkg.github.com/alterationx10/ursula"
)
ThisBuild / credentials += Credentials(
"GitHub Package Registry", // realm
"maven.pkg.github.com", // host
"alterationx10", // user
sys.env.getOrElse("GITHUB_TOKEN", "abc123") // password
)

We will use an environment variable GITHUB_TOKEN to provide our password. Note, that you could do the same thing for the user value.

4 SBT gotchas

This isn't an all-inclusive list, but just a couple of things to keep in mind.

GitHub packages only supports Maven structure, so we need to set publishMavenStyle to true. We will set out version schema to "early-semver", which keeps binary compatibility across patch updates within 0.Y.z until you hit 1.0.0.

The most important "gotcha" here is ThisBuild / publish / skip := true. Since I do not have a root project here, sbt will make a default one, and aggregate the projects into. This means that it will also try to publish a package named default! We can either define a root project as a placeholder, and configure it accordingly - or globally set the default to skip publishing, but then re-enabling it in the project we're looking to deploy. The latter is shown here.

ThisBuild / publish / skip := true
ThisBuild / publishMavenStyle := true
ThisBuild / versionScheme := Some("early-semver")

lazy val ursula = project
.in(file("ursula"))
.settings(
name := "ursula",
libraryDependencies ++= Seq(
"dev.zio" %% "zio" % "2.0.0-RC6"
),
fork := true,
publish / skip := false
)

Lights! Camera! GitHub Action!

Now that sbt has been included in the environment loaded into the setup-java action, this is easier than it's ever been. For any action, you can use that and just sbt <your task>.

For out case, we only want this to run when we create a release (which is a git tag action), so note the on: block.

We've set up our build.sbt file to use ENV variables that are automatically provided, but we also use the auto-generated ci token: GITHUB_TOKEN which is available automatically - that should be set in the env: block. If you wanted to use a personal access token, you could store and access the secret in the same way!

name: Publish Artifact on Release
on:
release:
types: [ created ]
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
jobs:
publish:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Set up JDK
uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: '17'
cache: 'sbt'
- name: Publish
run: sbt publish

To kick it off, you just need to create a release with a structured git tag. the hardest part is not mistyping for tag 🤣 The packages wills tart to show up on you repositories page, right below the "release" section.

Wrapping up

Now, you too can publish your scala artifacts to GitHub packages without relying on a pre-made sbt plugin! How exciting.

· 3 min read
Scala 2 => 3 Series

This is a part in an ongoing series dealing with migrating old ways of doing things from Scala 2 to Scala 3. It will cover the What's New in Scala 3 from the official site.

Check the Scala 2 => 3 tag for others in the series! For the repo containing all the code, visit GitHub. There are code samples for both Scala 2 and Scala 3 together, that are easy to run via scala-cli.

This post is centered around retroactively extending classes.

In Scala 2, extension methods had to be encoded using implicit conversions or implicit classes. In contrast, in Scala 3 extension methods are now directly built into the language, leading to better error messages and improved type inference.

Extensions are one of my favorite things to use in Scala. Personally, I like the ability to add functionality to "upstream" resources implicitly, but call that functionality explicitly. To me, it makes it less likely to break things during a refactor when you don't have to un-ravel a mysterious series of implicit def methods / conversions that you might not realize are being called.

The preface

For this example, let's say that we have some upstream domain model from a service we use but don't control.

case class UpstreamUser(id: Long, created: Instant, lastSeen: Instant)

In our service, we have a concept of when a user goes "stale" based on usage - but other services also have this notion, and differing beliefs about what conditions make a user stale - so we can't ask the upstream service to implement this for us on our model. Perhaps our model of what a stale user is changes over time as well.

Our conditions for a user going stale are:

  • A user was created over a year ago
  • A user hasn't been seen in the last week.

With that in mind, we could write some logic such as

import java.time.Instant
import java.time.temporal.ChronoUnit._
def isStale(created: Instant, lastSeen: Instant):Boolean = {
lastSeen.plus(7, DAYS).isBefore(Instant.now) &&
created.plus(365, DAYS).isBefore(Instant.now)
}

but calling that everywhere becomes a bit cumbersome, and it would be great if we could attach that functionality directly on UpstreamUser.

Scala 2

In scala 2, we can use an implicit class to achieve our goal. An implicit class should have only one constructor argument, of the Type that is being extended. It also needs to be housed in something, typically an outer object. This can make setting up implicit classes feel a bit "boilerplate-y".

object UpstreamUserExtensions {
implicit class ExtendedUpstreamUser(u: UpstreamUser) {
def isStale: Boolean = {
u.lastSeen.plus(7, DAYS).isBefore(Instant.now) &&
u.created.plus(365, DAYS).isBefore(Instant.now)
}
}
}

Now, with ExtendedUpstreamUser in scope to implicitly add our new functionality, we can (explicitly) call upstreamUserInstance.isStale as if it were on the model directly.

Scala 3

In Scala 3, it works much the same, but with less boilerplate. Instead of declaring an implicit class, you declare an extension: extension (u: UpstreamUser) where the argument matches the Type you're adding functionality to. This doesn't need to be housed in an object either!

The corresponding Scala 3 code would look like:

extension (u: UpstreamUser) {
def isStale: Boolean = {
u.lastSeen.plus(7, DAYS).isBefore(Instant.now) &&
u.created.plus(365, DAYS).isBefore(Instant.now)
}
}

and then we'll get the same upstreamUserInstance.isStale functionality as before.

Final Thoughts

Although the looks of the code have changed, if you're used to Scala 2 implicit classes, Scala 3 extensions will probably be a welcomed ergonomics change, with a familiar feel for usage.

· 4 min read
Scala 2 => 3 Series

This is a part in an ongoing series dealing with migrating old ways of doing things from Scala 2 to Scala 3. It will cover the What's New in Scala 3 from the official site.

Check the Scala 2 => 3 tag for others in the series! For the repo containing all the code, visit GitHub. There are code samples for both Scala 2 and Scala 3 together, that are easy to run via scala-cli.

This post is centered around the new way of passing implicit arguments to methods via using-clauses.

Abstracting over contextual information. Using clauses allow programmers to abstract over information that is available in the calling context and should be passed implicitly. As an improvement over Scala 2 implicits, using clauses can be specified by type, freeing function signatures from term variable names that are never explicitly referred to.

The preface

For this example, let's say that we have some interface that we're going to be passing around a lot, and that it could have multiple implementations.

trait BaseLogger {
def log[T](t: T): Unit
}

case class PrintLogger() extends BaseLogger {
def log[T](t: T): Unit = println(s"Logger result: ${t.toString}")
}

case class FancyLogger() extends BaseLogger {
def log[T](t: T): Unit = println(s"Ye Olde Logger result: ${t.toString}")
}

Scala 2

In Scala 2, we could write a method, and have our trait's implementation passed in as a separate implicit argument.

  def loggingOp[A,B](a: A, b: B)(implicit logger: BaseLogger): Int = {
val result = a.toString.map(_.toInt).sum + b.toString.map(_.toInt).sum
logger.log(result)
result
}

At this point, we could call our method by still passing the argument in explicitly

object Using_2 extends App {

val printLogger: PrintLogger = PrintLogger()
val fancyLogger: FancyLogger = FancyLogger()

loggingOp(40, 2)(printLogger)
loggingOp(40, 2)(fancyLogger)

}

However, if we define an instance of type BaseLogger in scope implicitly, then we don't need to pass it in as an argument every time! Of course, we still have the option to pass something in explicitly, if we don't want to use the instance that is in scope implicitly.


object Using_2 extends App {

val printLogger: PrintLogger = PrintLogger()
val fancyLogger: FancyLogger = FancyLogger()

loggingOp(40, 2)(printLogger)
loggingOp(40, 2)(fancyLogger)

// With an implicit of type BaseLogger in scope...
implicit val defaultLogger = printLogger

// ... I no longer need to pass it as an argument
loggingOp(true, false)
loggingOp(17, "purple")
// ... but I can still call implicit arguments explicitly!
loggingOp("car", printLogger)(fancyLogger)

}

Scala 3

In Scala 3, we don't use the implicit key word when defining a method - we now use using. A faithful port of the Scala 2 code above would look something like:

  // You can specify the name logger, but don't have to
def loggingOp_withParamName[A, B](a: A, b: B)(using logger: BaseLogger): Int = {
val result = a.toString.map(_.toInt).sum + b.toString.map(_.toInt).sum
logger.log(result)
result
}

The awesomeness of Scala 3 doesn't stop there, though, because you can define your methods by just declaring the type! In this case, we just summon an instance internally, and use reference to that.

There are only two hard things in Computer Science: cache invalidation and naming things.

Guess it's just invalidating caches now!

  def loggingOp[A, B](a: A, b: B)(using BaseLogger): Int = {
val logger = summon[BaseLogger]
val result = a.toString.map(_.toInt).sum + b.toString.map(_.toInt).sum
logger.log(result)
result
}

From here, our code works mostly the same - one caveat being that when explicitly passing arguments, you need to use the using keyword - where previously you didn't need to declare the values you were passing in were implicit. We're also declaring our BaseLogger in scope using alias givens

object Using_3 {

val printLogger: PrintLogger = PrintLogger()
val fancyLogger: FancyLogger = FancyLogger()

@main
def main = {

// We can still call things explicitly...
loggingOp(40, 2)(using printLogger)
loggingOp(40, 2)(using fancyLogger)

// .. but we have a new way of defining what type is in scope implicitly
// implicit val defaultLogger = printLogger // <- this would still work
given defaultLogger: BaseLogger = printLogger // <- but probably use this

loggingOp(true, false)
loggingOp(true, false)
loggingOp(17, "purple")
loggingOp("car", printLogger)(using fancyLogger)
}

}

Final Thoughts

Using clauses can be a bit more complex, but with the simple example outlined above - we have one less scary new thing, that we can mentally map back to our years of Scala 2 use!

· 11 min read

How to deploy Kubernetes meme

Premise

The example in this post is about using a kubernetes CustomResourceDefinition and Operator implemented with ZIO to simplify our lives as someone who made need to run a lot of infrastructure set up (dare I even say Dev/Ops).

The example is complete/functioning, but isn't the most robust solution for what it does. It is meant to be enough to work, and illustrate the concept with a solution to a made-up problem - but not exactly a model code base 👼

Let's dig in!

Hey, can you set me up a database?

Perhaps you're the one with the password/access to the database, or the only person nearby on the team that "knows SQL", but it's part of your daily life to set up databases for people. In between your coding work, you run a lot of the following type of code for people who need to access their own database from a kubernetes cluster:

CREATE DATABASE stuff;
CREATE USER stuff PASSWORD 'abc123';
GRANT ALL ON DATABASE stuff TO stuff;

Your hard work is then rewarded by remembering to set up a Secret for each database as well, so the user can easily mount it to their pods for access.

But, wait a minute - you've just picked up a nifty framework called ZIO, and have decided to automate a bit of you daily todos.

Enter ZIO

Let's create a SQLService that will set up a matching database and user:

trait SQLService {
def createDatabaseWithRole(db: String): Task[String]
}

// We're going to be lazy, and not use a Logger
case class SQLServiceLive(cnsl: Console.Service) extends SQLService {
override def createDatabaseWithRole(db: String): Task[String] = ???
}

We aren't running this so often that we need a dedicated connection pool, so let's just grab a connection from the driver, and use this neat new thing we've learned about called Zmanaged.

private val acquireConnection =
ZIO.effect {
val url = {
sys.env.getOrElse(
"PG_CONN_URL", // If this environment variable isn't set...
"jdbc:postgresql://localhost:5432/?user=postgres&password=password" // ... use this default one.
)
}
DriverManager.getConnection(url)
}

private val managedConnection: ZManaged[Any, Throwable, Connection] =
ZManaged.fromAutoCloseable(acquireConnection)

// We'll use a ZManaged for Statements too!
private def acquireStatement(conn: Connection): Task[Statement] =
Task.effect {
conn.createStatement
}

def managedStatement(conn: Connection): ZManaged[Any, Throwable, Statement] =
ZManaged.fromAutoCloseable(acquireStatement(conn))

What's a ZManaged?

ZManaged is a data structure that encapsulates the acquisition and the release of a resource, which may be used by invoking the use method of the resource. The resource will be automatically acquired before the resource is used and automatically released after the resource is used.

So a ZManged is like a try/catch/finally that handles your resources - but you don't have to set up a lot of boilerplate. A common pattern I've used in the past would be to use a thunk to do something similar. The (very unsafe, with no error handling) example below handles the acquisition and release of the connection + statement, and you just need to pass in a function that takes a statement, and produces a result.

def sqlAction[T](thunk: Statement => T): T = {
val url: String = "jdbc:postgresql://localhost:5432/?user=postgres&password=password"
val connection = DriverManager.getConnection(url)
val statement: Statement = connection.createStatement()
val result: T = thunk(statement)
statement.close()
connection.close()
result
}

def someSql = sqlAction { statement =>
// do something with statement
???
}

In the spirit of our thunk, we'll write a ZIO function that takes a Statement, a String (some SQL), and will execute it. We'll print the SQL we run, or log the error that falls out.

val executeSql: Statement => String => ZIO[Any, Throwable, Unit] =
st =>
sql =>
ZIO
.effect(st.execute(sql))
.unit
.tapBoth(
err => cnsl.putStrLnErr(err.getMessage),
_ => cnsl.putStrLn(sql)
)

Now with all of our pieces in place, we can implement our createDatabaseWithRole that will safely grab a Connection + Statement, run our SQL, and then automatically close those resources when done. It'll even hand back the random password generated:

override def createDatabaseWithRole(db: String): Task[String] = {
managedConnection.use { conn =>
managedStatement(conn).use { st =>
for {
pw <- ZIO.effect(scala.util.Random.alphanumeric.take(6).mkString)
_ <- executeSql(st)(s"CREATE DATABASE $db")
_ <- executeSql(st)(s"CREATE USER $db PASSWORD '$pw'")
_ <- executeSql(st)(s"GRANT ALL ON DATABASE $db TO $db")
} yield pw
}
}
}

😍 A thing ouf beauty! Now we can just make a simple ZIO program to call our new service, and call it a day!

val simpleProgram: ZIO[Has[SQLService], Nothing, Unit] =
SQLService(_.createDatabaseWithRole("someUser"))
.unit
.catchAll(_ => ZIO.unit)

Automate the Automation

j/k you still have to stop what you're doing to run this for people, and you still need to make the Secret! Wouldn't it be neat if we could have some sort of Kubernetes resource that allowed anyone to just update a straightforward file? What if we had something like:

apiVersion: alterationx10.com/v1
kind: Database
metadata:
name: databases
spec:
databases:
- mark
- joanie
- oliver

Well, it turns out we can have nice things! We can create a CustomResourceDefinition that will use that exact file as shown above! The following yaml sets up our own Kind called Database that has a spec of databases, which is just an array of String.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: databases.alterationx10.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: alterationx10.com
# list of versions supported by this CustomResourceDefinition
versions:
- name: v1
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
databases:
type: array
items:
type: string
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: databases
# singular name to be used as an alias on the CLI and for display
singular: database
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: Database
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- db

Since we don't want to run jobs manually, we can create an Operator that will watch for our CustomResourceDefinition , and take action automatically! With the zio-k8s library, these can be fairly straightforward to implement.

  val eventProcessor: EventProcessor[Clock, Throwable, Database] =
(ctx, event) =>
event match {
case Reseted() =>
cnsl.putStrLn(s"Reseted - will (re) add any existing").ignore
case Added(item) =>
processItem(item)
case Modified(item) =>
processItem(item)
case Deleted(item) =>
cnsl.putStrLn(s"Deleted - but not performing action").ignore
}

For our example program, we will always try and create the databases listed in the resources, and log/ignore the error if a database already exists on Added and Modified. We will also take the auto-generated password, and create a secret for it as well! We won't tear anything down on Deleted.

def processItem(item: Database): URIO[Clock, Unit] =
(for {
// Get all of our databases
dbs <- ZIO.fromOption(item.spec.flatMap(_.databases).toOption)
// For each database
_ <- ZIO.foreach(dbs) { db =>
(for {
_ <- cnsl.putStrLn(s"Processing $db...")
// Create things
pw <- sqlService.createDatabaseWithRole(db)
_ <- cnsl.putStrLn(s"... $db created ...")
// Put the generated PW in a k8s secret
_ <- upsertSecret(
Secret(
metadata = Some(
ObjectMeta(
name = Option(db),
namespace = item.metadata
.map(_.namespace)
.getOrElse(Option("default"))
)
),
data = Map(
"POSTGRES_PASSWORD" -> Chunk.fromArray(
pw.getBytes()
)
)
)
).tapError(e => cnsl.putStrLnErr(s"Couldn't make secret:\n $e"))
_ <- cnsl.putStrLn(s"... Secret created for $db")
} yield ()).ignore
}
} yield ()).ignore

def upsertSecret(
secret: Secret
): ZIO[Clock, K8sFailure, Secret] = {
for {
nm <- secret.getName
ns <- secret.getMetadata.flatMap(_.getNamespace)
existing <- secrets.get(nm, K8sNamespace(ns)).option
sec <- existing match {
case Some(_) => secrets.replace(nm, secret, K8sNamespace(ns))
case None => secrets.create(secret, K8sNamespace(ns))
}
} yield sec
}

That's about it! We now have the code we need to automate our daily drudgery!

Deploying

This example is targeted at deploying to the instance of Kubernetes provided by Docker, mainly so we can use our locally published docker image.

Auto generation of our CRD client

We will need the zio-k8s-crd SBT plugin to auto generate the client needed to work with our CRD. Once added, we can update our build.sbt file with the following, which points to the new CRD. With this in place, a compile step will generate the code for us.

externalCustomResourceDefinitions := Seq(
file("crds/databases.yaml")
)

enablePlugins(K8sCustomResourceCodegenPlugin)

Building a Docker image of our service

We'll use the sbt-native-packager SBT plugin to build the docker image for us. We'll need a more recent version of Java than what is default, so well set dockerBaseImage := "openjdk:17.0.2-slim-buster" and set our project to .enablePlugins(JavaServerAppPackaging). Now, when we run sbt docker:publishLocal, it will build and tag an image with the version specified in our build.sbt file that we can use in our kubernetes deployment yaml.

REPOSITORY      TAG            IMAGE ID       CREATED         SIZE
smooth-operator 0.1.0-SNAPSHOT a4e2c2025cba 2 days ago 447MB

Who doesn't love more YAML?

This section will go over the kubernetes yaml needed to deploy everything we need for our app.

We will create a standard Deployment of postgres, configured to have the super secure password of password 🤫. We will also create a Service to route traffic to it.

apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
labels:
app: postgres
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres
env:
- name: POSTGRES_PASSWORD
value: "password"
---
apiVersion: v1
kind: Service
metadata:
name: postgres
spec:
selector:
app: postgres
ports:
- port: 5432
targetPort: 5432
protocol: TCP

For deploying our Operator, we ultimately are going to set up a Deployment for it, but we're going to need a few more bells and whistles first. Our app will need the right permissions to be able to watch our CustomResourceDefinitions, as well as accessing Secrets - these actions are done by the ServiceAccount our pod runs under. We will create a ClusterRole that has the required permissions, and use a ClusterRoleBinding to assign the ClusterRole to our ServiceAccount.

A very useful kubectl command to check and make sure your permissions are correct is kubectl auth can-i ... command.

kubectl auth can-i create secrets --as=system:serviceaccount:default:db-operator-service-account -n default
kubectl auth can-i watch databases --as=system:serviceaccount:default:db-operator-service-account -n default

With all that in mind, we can use the following yaml to get our app up and running.

apiVersion: v1
kind: ServiceAccount
metadata:
name: db-operator-service-account
automountServiceAccountToken: true
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: db-operator-cluster-role
rules:
- apiGroups: [ "alterationx10.com" ]
resources: [ "databases" ]
verbs: [ "get", "watch", "list" ]
- apiGroups: [ "" ]
resources: [ "secrets" ]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: db-operator-cluster-role-binding
subjects:
- kind: ServiceAccount
name: db-operator-service-account
namespace: default
roleRef:
kind: ClusterRole
name: db-operator-cluster-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: db-operator
labels:
app: db-operator
spec:
selector:
matchLabels:
app: db-operator
template:
metadata:
labels:
app: db-operator
spec:
serviceAccountName: db-operator-service-account
containers:
- name: db-operator
image: smooth-operator:0.1.0-SNAPSHOT
env:
- name: PG_CONN_URL
value: "jdbc:postgresql://postgres:5432/?user=postgres&password=password"
---

Note: When deploying an operator "for real", you want to take care that only one instance is running/working at a time. This is not covered here, but you should look into Leader Election

Running the Example

You can view the source code on GitHub, tagged at v0.0.3 at the time of this blog post.

Assuming you have Docker/Kubernetes et up, you should be able to run the following commands to get an example up and running:

# Build/publish our App to the local Docker repo
sbt docker:publishLocal
# Deploy our CustomResourceDefinition
kubectl apply -f crds/databases.yaml
# Deploy postgres
kubectl apply -f yaml/postgres.yaml
# Deploy our app
kubectl apply -f yaml/db_operator.yaml
# Create Database Resource
kubectl apply -f yaml/databases.yaml

If you check the logs of the running pod, you should hopefully see the SQL successfully ran, and can also use kubectl to check for new Secrets!

➜ smooth-operator (main) ✗ kubectl logs db-operator-74f756c89c-x5f5b 
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Reseted - will (re) add any existing
Processing mark...
CREATE DATABASE mark
CREATE USER mark PASSWORD 'VCaHar'
GRANT ALL ON DATABASE mark TO mark
... mark created ...
... Secret created for mark
Processing joanie...
CREATE DATABASE joanie
CREATE USER joanie PASSWORD 'mdlQKB'
GRANT ALL ON DATABASE joanie TO joanie
... joanie created ...
... Secret created for joanie
Processing oliver...
CREATE DATABASE oliver
CREATE USER oliver PASSWORD 'vYODSt'
GRANT ALL ON DATABASE oliver TO oliver
... oliver created ...
... Secret created for oliver

Nice.

There you have it! After a day or two of set up, now you too can save tens of minutes every day!