funktionale-programmierung.de // Funktionale Programmierung

Functional Programming by Example: Refactoring a TypeScript/SolidJS Application | Part 1

Beat Hagenlocher — Wed, 04 Mar 2026 00:00:00 UTC

Have you ever looked at a piece of code and thought:

I think this works, but I can‘t quite convince myself it‘s correct?

The logic is relatively straightforward.
But the state is distributed across a handful of variables that could be combined in ways that don‘t make sense.
And as soon as you start working on it, you find yourself mentally tracing through every code path to make sure none of the illegal variable combinations can happen.

Whenever this happens, one thing is part of the problem: The model of the domain was only ever implicitly there, in the head of the programmer, and not written down in code.

In this series, I want to show you how to formulate those models in code, in TypeScript, so that the code gets easier to understand for everyone not currently writing the code (colleagues, but also future you!). We‘re going to take a small but realistic application and refactor it from a hidden-model style to an explicit-model style.

We‘ll start out with a small but realistic application—a web audio player—in a hidden-model style and rework it. Along the way, you‘ll learn how to:

make illegal states unrepresentable using algebraic data types
how to push side effects to the boundary of your application
and how to use reducers and pure functions to make state transitions explicit and testable.

(among others)

Why an Audio Player?

Mostly to keep the mental workload small: You already have a good mental model of how an audio player should work, and whatever we‘re going to add to it, it‘s still going to be relatively simple. These properties make it a good example application for learning the techniques, and not the domain, of this tutorial.

Also, it‘s still a barebones useful thing, and not the, say, standard counter example.

Why SolidJS?

It looks like React: Its components are functions that produce TSX, the TypeScript variant of JSX (which is the JavaScript-interpolation-in-HTML language React uses)

But it‘s simpler than React: It uses signals for everything, which means there are never any full component rerenders, for example.

It‘s also smaller (both in API surface area and shipping size) and faster, but that‘s just a nice side effect :)

Approach

We‘re starting out writing the player in ‚normal‘, hidden-model SolidJS style.

Then, we‘re going to take a look at the hidden model, make it explicit, and transform the whole codebase accordingly.

After this, I‘m going to show you avenues for improvement: When and where to use functional data structures, more precisely typed records, and so on.

Audio Player: The Hidden-Model Version

Let‘s start with the barebones component:

import { createSignal, For, Show } from 'solid-js'
export const ImperativeAudioPlayer = () => {
  return (
    <div class="p-5">
      <h2>Imperative Audio Player</h2>

      <For each={tracks}>
        {(track) => <button onClick={() => playTrack(track)}>Play {track.title}</button>}
      </For>

      <div class="mt-4 p-2 border-[1px] border-solid border-red-500">
        <Show when={currentTrack()} fallback={<p>No track selected</p>}>
          <h3>Now: {currentTrack().title}</h3>

          <Show when={isLoading()}>
            <p>Loading...</p>
          </Show>

          <p>Time: {currentTime().toFixed(2)}s</p>

          <button onClick={togglePlay}>{isPlaying() ? 'Pause' : 'Resume'}</button>
        </Show>
      </div>
    </div>
  )
}

Add in the signals:

const [isPlaying, setIsPlaying] = createSignal(false)
const [isLoading, setIsLoading] = createSignal(false)
const [currentTrack, setCurrentTrack] = createSignal<any>(null)
const [currentTime, setCurrentTime] = createSignal(0)

The audio element:

const audio = new Audio()

The callback functions for the elements:

const playTrack = (track: any) => {
  setCurrentTrack(track)
  setIsLoading(true)
  setIsPlaying(false)

  audio.src = track.url
  audio.play().catch((err) => {
    console.error('Playback failed', err)
    setIsPlaying(false)
  })
}

const togglePlay = () => {
  if (audio.paused) {
    audio.play()
    setIsPlaying(true)
  } else {
    audio.pause()
    setIsPlaying(false)
  }
}

Now, there‘s only two things left:

Initializing the audio element with the correct callbacks:

onMount(() => {
  audio.oncanplay = () => {
    setIsLoading(false)
    setIsPlaying(true)
  }

  audio.ontimeupdate = () => {
    setCurrentTime(audio.currentTime)
  }
})

And actually adding in tracks:

const tracks = [
  {
    id: '1',
    title: 'Lo-Fi Beats',
    url: 'https://www.soundhelix.com/examples/mp3/SoundHelix-Song-1.mp3',
  },
  {
    id: '2',
    title: 'Synthwave',
    url: 'https://www.soundhelix.com/examples/mp3/SoundHelix-Song-2.mp3',
  },
]

This should complete to something like this: SolidJS playground, Github Permalink

Making Illegal States Unrepresentable: Modeling Intent

Right now, isPlaying, isLoading, currentTrack and currentTime are independent values. That‘s at least what the code suggests to the reader on first sight.

The names of the variables (and our mental model of what an audio player should be doing) suggest that they‘re not independent.

Let us think about this:
Most probably, the code should move from a state where there‘s no song selected (isLoading: false and currentTrack: null) to a loading state (isLoading: true with a currentTrack), and then to playing a track (isPlaying: true).

(And after mental tracing of the code paths, we see that is actually the case—yay!)

The problem with this is the following:

The signals‘ names here only hint at a mental model the original programmer (that‘s us!) tried to implement. And the current implementation adheres to this intent. (This is another way of saying that the code is correct.)

The question is what happens in the future: Every time someone is reading the code again (that‘s probably us), they‘re going to do this mental gymnastics to arrive at the correct mental model of the intent.

And there is a risk in there:

Since our intent is nowhere explicitly written down, it‘s easy to make a change so that the system arrives in a state like {isPlaying: true, isLoading: true, *} (which isn‘t part of the states we intended and therefore should be impossible to arrive in).
This is bad.

So let‘s try and formalize our intent into a model, and write this model down explicitly (in words, first).

Writing the Model Down

Our Audio player should be in no more than these kinds of states:

An Initial state ‚kind‘: When the song‘s not playing, and hasn‘t started loading yet. (That would correspond to {isPlaying: false, isLoading: false, currentTrack: null, currentTime: null}.)
A Loading state ‚kind‘: We have a track selected, but it hasn‘t loaded yet. (corresponds to {isPlaying: false, isLoading: true, currentTrack: someTrack, currentTime: null})
A Playing state ‚kind‘: When the song is done loading, and playing. (corresponds to {isPlaying: true, isLoading: false, currentTrack: someTrack, currentTime: someTime})
A Paused state ‚kind‘: When we‘ve paused the song. (corresponds to {isPlaying: false, isLoading: false, currentTrack: someTrack, currentTime: someTime})

Sidenote: Every state ‚kind‘ has a possible very large number of states, like the number of all tracks one might play—but since we don‘t need to disambiguate between the different songs, this doesn‘t have any effect on the complexity of the model.

Okay, this model sounds sensible: Let‘s implement it in TypeScript.

Making Illegal States Unrepresentable: Implementation

Writing the Model Down (in Code)

The way we‘re going to implement this model is by using TypeScript‘s tagged unions. (The ‚tag‘ here being the additional ‚type‘ key in the object notation)

These are the four legal states:

export type Track = { id: string, url: string, title: string }
export type Time = number

export type Initial = { type: 'Initial' }
export type Loading = { type: 'Loading', track: Track }
export type Playing = { type: 'Playing', track: Track, currentTime: Time }
export type Paused = { type: 'Paused', track: Track, currentTime: Time }

And our whole AudioPlayer State is then a Union of those types:

export type State = Initial | Loading | Playing | Paused

Explicit Model Transitions

Let‘s start using this State model in our audio player. For this, we need to add the initial state:

const [state, setState] = createSignal<State>({ type: 'Initial' })

Then, we can visit each usage of isPlaying, isLoading, currentTrack and currentTime, and replace them with their State equivalent.

I find that easiest to refactor this from the initial state on:

After loading the page, we‘re in the Initial state.
Then, as soon as we click playTrack, we should go over to the Loading state
Then, we trigger the mp3 load via setting audio.src

  const playTrack = (track: Track) => {
    setState({ type: 'Loading', track })

    audio.src = track.url
  }

As soon as the source is loaded, oncanplay fires:

    audio.oncanplay = () =>
      setState((prev: State) => {
        if (prev.type === 'Loading') {
          return { type: 'Playing' as const, track: prev.track, currentTime: 0 }
        } else {
          return prev
        }
      })

This should make our audio.play() in playTrack succeed:

  const playTrack = (track: Track) => {
    setState({ type: 'Loading', track })

    audio.src = track.url
    audio.play().catch((err) => {
      console.error('Playback failed', err)
      setState({ type: 'Paused', track, currentTime: 0 })
    })
  }

After which our ontimeupdate triggers regularly:

    audio.ontimeupdate = () =>
      setState((prev: State) => {
        if (prev.type === 'Playing') {
          return { ...prev, currentTime: audio.currentTime }
        } else {
          return prev
        }
      })

Now, we should be able to togglePlay:

  const togglePlay = () =>
    setState((prev: State) => {
      if (prev.type === 'Paused') {
        audio.play()
        return { ...prev, type: 'Playing' }
      } else if (prev.type === 'Playing') {
        audio.pause()
        return { ...prev, type: 'Paused' }
      } else {
        return prev
      }
    })

And with that, we have our state change logic.

TypeScript Type Narrowing in SolidJS

We also need to update all the usages of our state in the JSX, and there, we encounter a problem:

When doing something like <Show when={state().type === 'Loading'}> ... </Show> in SolidJS, TypeScript can‘t correctly narrow the type of state() to the Loading state.

(And we should use the Show component for correctly working reactivity in SolidJS, not just some {state().type === 'Loading' && ...})

To circumvent this, we need to use the callback children version Show provides, and then provide narrowed type annotations. This looks like this:

  const trackState = () => {
    if (state().type === 'Loading' || state().type === 'Paused' || state().type === 'Playing') {
      return state() as Loading | Playing | Paused
    } else {
      return undefined
    }
  }

        <Show when={trackState()} fallback={<p>No track selected</p>}>
          {(ts) => ( // <- the narrowed state gets used here.
            <>
              <h3>Now: {ts().track.title}</h3>

              <Show when={state().type !== 'Loading'} fallback={<p>Loading...</p>}>
                <p>Time: {(state() as Playing | Paused).currentTime.toFixed(2)}s</p>

                <button class="p-2 bg-zinc-200 m-2 cursor-pointer" onClick={togglePlay}>
                  {state().type === 'Playing' ? 'Pause' : 'Resume'}
                </button>
              </Show>
            </>
          )}
        </Show>

Our whole JSX now looks like this:

  return (
    <div class="p-5">
      <h2>Audio Player: Illegal States Are Unrepresentable</h2>
      {JSON.stringify(state())}

      <For each={tracks}>
        {(track) => (
          <button class="p-2 bg-zinc-200 m-2 cursor-pointer" onClick={() => playTrack(track)}>
            Play {track.title}
          </button>
        )}
      </For>

      <div class="mt-4 p-2 border-[1px] border-solid border-red-500">
        <Show when={trackState()} fallback={<p>No track selected</p>}>
          {(ts) => (
            <>
              <h3>Now: {ts().track.title}</h3>

              <Show when={ts().type !== 'Loading'} fallback={<p>Loading...</p>}>
                <p>Time: {(ts() as Playing | Paused).currentTime.toFixed(2)}s</p>

                <button class="p-2 bg-zinc-200 m-2 cursor-pointer" onClick={togglePlay}>
                  {ts().type === 'Playing' ? 'Pause' : 'Resume'}
                </button>
              </Show>
            </>
          )}
        </Show>
      </div>
    </div>
  )

Full code for this version here: SolidJS playground, Github

With this, we have achieved our goal: The model is explicitly written down in code, and it‘s not possible to represent an illegal state anymore.

What is possible (and easy) right now, however, is making a mistake in the state transitions (since they‘re distributed over the whole codebase).

In the next part, we‘re going to use the reducer pattern and pull our state transitions together into one function called reduce:

export const reduce = (oldState: State, action: Action): State => {
  // ...
}

See you there!

The program für BOB 2026 on March 13 is up!

Michael Sperber — Fri, 12 Dec 2025 00:00:00 UTC

The program for BOB 2026, Active Group‘s in-house conference, is set: On Friday, March 13, 2026, BOB - appropriately the thirteenth edition - will take place — as last year at the Scandic Hotel Potsdamer Platz.

We are proud of the program, although we had to reject many other great submissions to create it.

The opening keynote at BOB will be delivered by Stefan Kaufmann (stk) — it‘s about Digital Sovereignty.

As usual, there are four tracks — two tracks with a total of 16 talks, two tracks with a total of 8 tutorials.

Registration is open — the early-bird discount runs until January 16, 2026.

Our goal is always to make the conference contributions accessible to as many participants as possible. It is therefore possible to fill the entire day with English-language talks and tutorials. There are also some German-language contributions.

Talks

This year, what‘s striking is what‘s missing: talks about „AI“. Since all other developer conferences lately have been quite overrun with the topic, BOB 2026 is an opportunity for a break - and a reminder that there is still IT beyond AI.

As always strongly represented in the program: functional programming. Topics include OCaml (twice!, Scala, Java, functional software architecture and functional programming with SwiftUI.

Exotic programming languages (even by BOB standards) will also be present.

Also no surprise - formal methods with Lean, formal specifications and tests and refinement types.

We‘ll also see talks about accessibility, UI development, database joins, Domain-Driven Design and reactive systems.

Hopefully there‘s something for everyone - at least if you don‘t absolutely need „AI“.

Tutorials

The BOB tutorials are each 90 minutes long:

Specific technologies are represented with Agda, Scheme for 3D graphics, TypeScript, Multicore OCaml, Haskell as well as Mutiny and Quarkus.

We‘ll also talk about illegal states (and how to avoid them) and look forward to a tutorial on accessibility.

Registration

Registration is open online. Early-bird discount run until January 16, after that it becomes somewhat more expensive. There are also discounts and free tickets for underrepresented groups.

On the Testability of Pure Functions

Markus Schlegel — Thu, 16 Oct 2025 00:00:00 UTC

While working on the paper „Evolution of Functional UI Paradigms“, published at the FUNARCH workshop of ICFP 2025, I had a thought about pure functions that I would like to briefly explain here. As an argument in favor of pure functions, functional programmers often cite better testability: pure functions require no complicated test setup or mocks, are deterministic, parallelizable, etc. In tendency, this is certainly correct. Pure functions are often better suited for testing than their impure counterparts. However, this connection is not a neccessary implication in either direction: There are pure functions that are hard to test, and there are impure functions that are easy to test. Thus purity itself cannot be the substance that ensures a piece of program is properly testable. But then what is this substance?

Before I attempt an answer, I want to briefly explain why the link between purity and testability is not mandatory.

There are impure functions that are perfectly testable

An impure function is one whose behavior cannot be fully determined only from a description of the relation between input and output values. This can happen, for example, when the inputs and outputs are not pure (mathematical) values but mutable objects—i.e., memory locations to which different values can be assigned.

public class Counter {
    private int value;

    public Counter() {
        this.value = 0;
    }

    public int getValue() {
        return value;
    }
    
    public void inc() {
        this.value = this.value + 1;
    }
}

...

Counter counter = new Counter();
counter.inc();
assert counter.getValue() == 1;

Here, the variable counter refers to a Counter, which is a mutable storage of int values. The method inc, which implicitly takes such a storage (this) as an argument, is definitely not pure. Yet it is simple to test. We expect that inc increments the counter, and we can verify exactly that with three simple lines of test code. From a software architecture view, mutable state has other issues, but at least in this simple example, testability is unproblematic.

There are pure functions that are poorly testable

On the other hand, pure functions are not necessarily easy to test. At Active Group we write a lot of UI code with the reacl-c ClojureScript library. That code is composed almost exclusively of pure functions, yet there are significant parts of our GUI code that are not tested automatically. That‘s not because we‘re lazy (writing tests makes programming easier!), but because UI code is often inherently hard to test. Consider this example in ClojureScript:

(defn view [temp]
  (dom/div
    {:style (when (too-hot? temp)
               {:background "red"})}
    (temperature->string temp))

(assert-equal (view 22) (dom/div "22")

(assert-equal (view 183)
  (dom/div {:style {:background "red"}} "183"))

The view function maps „temperature“ values to a UI representation. The temperature value should appear as a number in the UI. In addition, „too high“ temperatures (whatever that means) should be emphasized more strongly. Here we chose to implement the concept of „emphasis“ by applying a red background.

The function view is pure – no changes or side effects occur – and we can write simple tests for it, as shown above. However, those tests are poor, because they check an aspect of the implementation: they require a red background and allow no alternative implementation of the same idea of emphasis. If later we decide that a nicer implementation would be a red border or a small red exclamation mark, we would have to change both implementation and tests. Tests that must be constantly adapted to implementation are almost worthless.

Good tests check whether an implementation satisfies its specification – and that is very hard in this example. In natural language, we might express the specification for view like this: Show the temperature as text and emphasize too-high temperatures. But how can this specification be formalized so that it can be verified by automated tests?

The substance that guarantees testability

I cannot answer this question here. On the contrary: I assert that some aspects a computer program handles are simply not formalizable – and so not automatically testable. With this article I only want to show that pure functions are not a guarantee for good testability. Instead, I claim that this relationship is mediated by something else. The temperature display example sheds light on what that something else is: precise and simple specifications.

First, one must make clear that every piece of program is written toward a goal that is not contained in the code itself. In other words: code is never an end in itself. If one takes care, one doesn’t merely negotiate those goals in endless meetings, but writes them down. The result is specifications. These specifications are more valuable when they are simple (yet sufficient) and precise.

Pure functions start in pole position regarding the simplicity of their specifications. The absence of mutations and side effects means that a pure function can be characterized solely by the relationship between its inputs and outputs. With impure functions there is more room for nonsense. As programmers, we must restrain that nonsense by discipline. The Counter example above shows that this is indeed possible. That counter has a very simple and precise specification, and so it is also easy for us to write a proper test.

On the other hand, the temperature example also shows that testability can fail even with pure functions, namely when the specification is imprecise. Likewise, one could imagine examples with precise but complex specifications. Again, testability would suffer regardless of (im-)purity.

In our FUNARCH paper we suggest one way to deal with those imprecise specifications. Instead of giving up and writing no UI tests at all, we propose addressing the problem with classic software architecture craftsmanship by separating responsibilities. Concretely, we suggest the functional Model-View-ViewModel (MVVM) pattern: aspects of UI code that can be precisely specified go into a separate model (consisting of immutable data and pure functions), and so become testable. The genuinely difficult-to-specify aspects of the user interface are pushed into a minimal and – thanks to the reacl-c programming model – loosely coupled area of the application, thereby minimizing the effort of manual (and thus costly) testing.

The Call for BOB 2026 is out!

Michael Sperber — Tue, 30 Sep 2025 00:00:00 UTC

On March 13, 2026, BOB, Active Group‘s conference about the best in software development, will take place — once again in Berlin and once again at the Scandic Berlin Potsdamer Platz.

The Call for Contributions is open. Send us your proposal for a talk or tutorial (by November 17, 2025) — the program committee is looking forward to it!

BOB 2026

Every year, BOB is about techniques and technologies that represent the best in their respective fields for developers — appropriate, particularly well-suited, and preferably beyond the mainstream. Here‘s a list of our primary topics:

functional programming
persistent data structures and databases
event-based modelling and architecture
„fancy types“ (dependent types, gradual typing, linear types, …)
formal methods for correctness and robustness
abstractions for concurrency and parallelism
metaprogramming
probabilistic programming
math and programming
controlled side effects
program synthesis
Ai beyond vibecoding and chatbots
linked data
symbolic AI
next-generation IDEs
effective abstractions for data analytics

We‘ve added three new items related to „AI“. If you‘re submitting in this area, you should pay attention to BOB‘s general goal — specifically, that the techniques applied are appropriate and particularly well-suited. The topic areas „Linked Data“ and „Symbolic AI“ are also meant to open a small door to what used to be called „AI“ before neural networks and LLMs came to dominate the field.

Also of interest are submissions on special challenges in software development in our time:

bias in machine-learning systems
digital transformation in difficult settings
accessibiltity
systems with critical reliability requirements
ecologically sustainable software development

BOB is also always open to new topics — just mind that they should always be appropriate and particularly well-suited.

In 2026, we are once again offering speaker grants. The speaker grants are intended to support groups underrepresented at BOB. This particularly includes women and speakers who would not be able to attend BOB for financial reasons. We will also once again offer free childcare on-site. These grants subsidize travel to and accommodation in Berlin.

So send us your proposal for a talk or tutorial! You can do so in German or English. We expect to have two talk tracks and two tutorial tracks again.

Is there a tutorial or topic you would like to see at BOB that has been missing so far? We‘re happy to receive your suggestions and wishes: as comments on this blog post or by email to konferenz at bobkonf dot de.

Spring Boot with Scala

David Frese — Thu, 06 Feb 2025 00:00:00 UTC

The Spring Framework and especially Spring Boot, are very popular and widely used for programming applications in Java. For Scala, there are other popular frameworks, particularly for programming web servers, such as the Play Framework.

In this small series of articles, we want to explore whether and to what extent we can program functionally in Scala while still using Spring Boot.

The Spring Framework consists of a whole range of elements, some of which form a kind of central core while others, building on the core, have more of a library character.

Dependency Injection

Among the central elements is a mechanism for dependency injection, also called auto-wiring. In various highly configurable ways, this finds classes that appear as arguments or members in other classes or methods and instantiates them automatically at runtime.

Annotations

Spring and Spring Boot also rely heavily on annotations to mark classes and methods for various purposes and to add code to them. Alternatively, this can at least partially be done via a configuration file (XML or YAML), but that isn‘t very popular since these files can quickly become very large.

Spring Boot

Spring Boot is a collection of libraries designed to make handling and getting started with the Spring Framework easier. This happens primarily through so-called starters, which bring many defaults with them and are intended to make it easier to implement typical use cases.

A list of starters can be found here.

The focus is primarily on web servers and database connections.

Example Application

In a small example, we‘ll now explore what this looks like in practice with Scala and as much functional programming as possible.

We‘ll implement a very simple (and morbid) game in which armadillos venture onto a highway and may get run over. Based on a functional core, we‘ll use Spring Boot to implement a REST API for the game.

Let‘s start with the functional core. A Dillo (short for Armadillo) has a name and a health level. When it gets run over, the health level decreases depending on the speed. Additionally, it can say hello as long as it‘s still alive:

case class Dillo(name: String, health: Int)

object Dillo {
  def runOver(d: Dillo, speed: Int): Dillo = {
    d.copy(health = Math.max(0, d.health - speed))
  }

  def speak(d: Dillo): String = {
    if (d.health > 0)
      s"${d.name} says Hi!"
    else
      s"${d.name} is dead"
  }
}

Then we define a highway with armadillos that are on it, and a method that lets the Dillos say something:

case class Highway(dillos: Map[UUID, Dillo]) {
  def report(): Iterable[String] = {
    for (_id, d) <- dillos
      yield Dillo.speak(d)
  }
}

We abstract functions that do something with the highway as Highway.Op - a so-called arrow. One of them is driveAlong, which runs over all armadillos once. A second one adds a new armadillo to the highway:

object Highway {
  type Op = Highway => Highway

  val empty: Highway = Highway(Map.empty)

  def driveAlong(speed: Int): Op = { highway =>
    highway.copy(dillos = highway.dillos.view.mapValues(Dillo.runOver(_, speed)).toMap)
  }

  def spawn(id: UUID, dillo: Dillo): Op = { highway =>
    highway.copy(dillos = highway.dillos + (id -> dillo))
  }
}

So much for the functional core of our application. Now let‘s move on to the REST API.

For our example, it‘s sufficient to include the Spring Boot starter for web applications: spring-boot-starter-web. In SBT we can do this as follows:

libraryDependencies += "org.springframework.boot" % "spring-boot-starter-web" % "3.4.2"

Then we first define a class that represents our application:

import org.springframework.boot.autoconfigure.SpringBootApplication

@SpringBootApplication
class DemoApp {}

The SpringBootApplication annotation implicitly brings a whole series of additional annotations with it. In particular, the so-called ComponentScan, which causes the Spring Framework to find classes in the application‘s classpath for dependency injection (so-called components). Also the EnableAutoConfiguration annotation from Spring Boot, which sets certain defaults depending on the libraries in the classpath and the presence (or absence!) of certain classes in the namespace of the application class. Therefore, just by adding a class or annotation, or by adding a library, the behavior of the application can change.

As the entry and starting point of our application, we now define a main method for the companion object of DemoApp:

object DemoApp {
  def main(args: Array[String]): Unit = {
    import org.springframework.boot.SpringApplication

    SpringApplication.run(classOf[DemoApp], args: _*)
  }
}

This is enough to have an executable application that first outputs a wonderful Spring logo:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::                (v3.4.2)

followed by some log messages showing that Spring Boot by default starts an embedded Apache Tomcat web server on port 8080 and registers our application in it as a servlet.

However, this is initially still empty. So let‘s define a so-called Controller to process HTTP requests. The easiest way to do this is with a class using the RestController annotation:

import org.springframework.web.bind.annotation.{GetMapping, RestController}

@RestController
class DemoController {
  @GetMapping() def index: String = "Hello world"
}

Spring Boot finds this class and binds it accordingly into the servlet. If we now start the application and send a request to "/", we get a corresponding response:

$ curl http://localhost:8080/
Hello World

However, to make our game „playable“ via such a RestController, we need to figure out how to manage the state of the game (or of the highway) over time. Since Spring Boot unfortunately doesn‘t offer any functional interfaces for this, we have to work imperatively. What we can do, however, to delay this decision as long as possible, is define a trait:

trait IAppState {
  def get: Highway
  def update(op: Highway.Op): Unit
}

We can then use this in our controller to return the highway report as a response to "/":

@RestController
class DemoController(state: IAppState) {
  @GetMapping(value = Array("/")) def index: String =
    state.get.report().mkString("\n")
}

If we now start our application, Spring complains that it can‘t find an implementation of IAppState. So let‘s define an implementation that stores the state in an AtomicReference:

import org.springframework.stereotype.Component

@Component
class MemAppState extends IAppState {
  import java.util.concurrent.atomic.AtomicReference
  
  private val highway: AtomicReference[Highway] = AtomicReference[Highway](Highway.empty)

  def get: Highway = highway.get()

  def update(op: Highway.Op): Unit = {
    highway.updateAndGet({ highway => op(highway) })
  }
}

Through the Component annotation, we make the class discoverable for Spring. The framework creates an instance of our MemAppState class when the application starts in order to create the controller with it. If multiple classes exist that implement IAppState, Spring aborts the application with a corresponding error, since it can‘t decide which one to use.

So that something can actually be changed on the highway via the HTTP interface, we now define a request to add an armadillo and one to drive along the highway:

@RestController
class DemoController(state: IAppState) {
  @GetMapping() def index: String =
    state.get.report().mkString("\n")

  @PostMapping(Array("/dillo")) def dillo(name: String): UUID = {
    val id = UUID.randomUUID()
    state.update(Highway.spawn(id, Dillo(name, health = 100)))
    id
  }

  @PostMapping(Array("/drive")) def drive(speed: Int): Unit = {
    state.update(Highway.driveAlong(speed))
  }
}

The default path of the mappings is always „/“, regardless of what the method is called. The subpaths „/dillo“ and „/drive“ must be explicitly declared as Array in Scala.

A big pitfall exists in Scala 3 before version 3.6, where you can write the annotation as it appears in many Java tutorials for Spring:

@PostMapping("/dillo")

However, this code compiles to something different than in Java and sets the name of the annotation instead of the value. You‘ll notice this either when the requests don‘t work or when Spring complains about multiply declared paths.

Now we can play our game with CURL!

$ curl -X POST -d "name=Sammy" localhost:8080/dillo
"5ee5db86-8906-4a9e-a37c-543d69015455"

$ curl -X POST -d "speed=80" localhost:8080/drive

$ curl -X POST -d "name=Mona" localhost:8080/dillo
"d39f99ee-6405-4bce-8ee3-0157b1c4adcb"

$ curl -X POST -d "speed=90" localhost:8080/drive

$ curl localhost:8080/
Sammy is dead
Mona says Hi!

The names of the parameters of the two methods dillo and drive are by default also the names of the form parameters in the POST request.

Conclusion

So we were able to create a web server with Spring Boot and connect our functional application core to it. That‘s not bad at all, and in some situations it may be a good alternative to the Play Framework or other Scala libraries.

In a follow-up article, we‘ll explore whether and how we can use Spring Data to store the application state in a database.

Data Modeling with Sums and Products

Michael Sperber — Mon, 25 Nov 2024 00:00:00 UTC

This is an English translation of the German version of this post.

Data Modeling is often an underappreciated aspect of software architecture, yet it plays a crucial role in achieving not only functional but also usability and maintainability goals. Poor data models and poorly integrated data models can greatly hinder architecture work. Consequently, data modeling — particularly of a project‘s core information — should be considered a fundamental responsibility of software architecture.

This is specifically relevant for the iSAQB Foundation training, which has just gained a new learning goal on data modeling.

This article examines two basic tools for good data models: sums and products. These concepts are known under a variety of names, depending on context, community, and programming language. Products are also known as records, structs, data classes, tuples, or and data, and sums are known as discriminated union, disjoint union, union, mixed data, and or data.

Sums and products have their roots in algebraic data types, which are well known from functional programming languages like Haskell or OCaml. However, the underlying concept is independent from a particular programming language. In many years of experience in software architecture and development, we have used this concept as a valuable tool for all kinds of data modeling tasks, in various programming languages and contexts.

This article explains the simple concept of sums and products. Further, it shows how to encode a simplified real-world scenario using sums and products in modern programming languages (Java, Python, Haskell, Kotlin, C#, Racket, Clojure, Scala, F#, Swift, Rust, Typescript).

Scenario

Our scenario is based on long-standing experience with a large, commercial software system that implements a health information system to be used by hospitals. Clearly, we have greatly simplified the scenario and omitted many details.

A health information system should provide information about the medication of a patient. In our simplified scenario, a medication consists of the name of a drug and its dosage. There are two different kinds of dosages, depending on whether the drug is administered orally via tablets or intravenously via an infusion.

The dosage for tablets specifies the number of tablets to be taken in the morning, at midday, and in the evening. For example, the dosage 1-0-2 specifies that one tablet should be taken in the morning, no tablet at midday, and two tablets in the evening.

The dosage for infusions specifies how fast the infusion should flow into the body (in milliliters per minute) and how long the infusion should run (in hours). For example, the dosage 1.5ml/min for 2h specifies that the infusion should run for 2 hours at the speed of 1.5ml per minute.

Products and Sums

Let‘s look at the structure of the data involved:

Medication consists of the drug name and the dosage.
Dosage is either a dosage of tablets or of infusions.
Dosage of tablets consists of the amount of tablets to be taken at morning, midday, and evening.
Dosage of infusion consists of the speed (ml/min) and the duration (h).

You can see a few recurring words here: Specifically, the description of medication and the two different dosages uses „consists of“ and the word „and“, whereas the description of dosage uses „or“. Different wordings are possible (e.g. „has attributes“ for the former and „is one of the following“ for the latter), but the wordings always describe the two sorts of data that are fundamentally different - one is „and data“, the other is „or data“. As stated above, these two concepts go by different names, but the most common ones are products (for „and data“) and sums (for „or data“).

A product has several, fixed attributes.
A sum has several, distinct alternatives.

Code for Sum and Products

Here is a translation of the data descriptions into Java code:

public record Medication(String drugName, Dosage dosage) {}

public sealed interface Dosage {
    record Tablet(int morning, int midday, int evening) implements Dosage {}
    record Infusion(double speed, int duration) implements Dosage {}
}

Java supports products through record classes, and sums through the combination of sealed interfaces and classes implementing them. (You might have noticed that this code is not „classic Java“ and uses fairly recent features - we‘ll get to that.)

Programming languages differ in how they support sums and products. Here is Python, for example:

from dataclasses import dataclass

@dataclass
class TabletDosage:
    morning: int
    midday: int
    evening: int

@dataclass
class InfusionDosage:
    speed: float
    duration: int

type Dosage = TabletDosage | InfusionDosage

@dataclass
class Medication:
    drugName: str
    dosage: Dosage

As you can see, products are similar to Java, only that records are called data classes in Python. Sums, however, are not realized via interface implementation, but via a separate definition of Dosage where | denotes „or“.

The data in this example fits a pattern extremely common in data modeling: Is the data of this kind or that kind, and depending on the kind, data has attributes A1 and A2 or B1 and B2. In our terminology, we call this a sum of products. (Clearly, more than two kinds and attributes are possible.)

Typed functional languages have a feature that directly implements sum-of-products - so-called algebraic data types. Here is code in the functional language Haskell:

data Dosage
  = TabletDosage { morning :: Int, midday :: Int, evening :: Int }
  | InfusionDosage { speed :: Double, duration :: Int }

data Medication = Medication { drugName :: String, dosage :: Dosage }

Dosage is the sum of the products TableDosage (with attributes morning, midday, and evening) and InfusionDosage (with attributes speed and duration). Medication can be seen as a sum with only one alternative, namely the product Medication (with attributes drugName and dosage).

Sums and Products Gone Wrong

Even though the distinction between sums and products is usually quite clear from a description of the information to be represented in the software system, its implementation goes wrong surprisingly often.

One reason for this is that some popular languages, mechanisms and formalisms do not have direct support for sums.

Take SQL, for example, and imagine you‘d have to store medications in a table. A table/relation in SQL has a fixed set of columns, and we have to somehow map the information about dosages into a fixed format. One way to do it would be to make columns for all possible attributes, like so:

CREATE TABLE medications(
	drugName VARCHAR(255) NOT NULL,
	dosageKind int NOT NULL, -- 1 for tablet, 2 for infusion
	morning int,
	midday int,
	evening int,
	speed double,
	duration int)

Here, the dosageKind column is a tag indicating which case of the sum it is. If it‘s 1, the row is supposed to represent a tablet dosage, and we‘d expect morning, midday, and evening to be non-null integers. Presumably, speed and duration should be null. Conversely for the other case.

This is a quite indirect encoding of a sum as a product, with the help of nullable types. This encoding poses significant risks of being misused: What if dosageKind is 1, but morning is null, and speed is 5? Everyone who has been around real-world SQL databases has seen rows like that, and the ensuing messiness and architecture problems.

Of course, tables in a relational database are just an external representation of the data, and an application is free to convert between a „proper“ data model in the software itself and its relational encoding. And it well should, be it via explicit code or via careful use of data transfer objects.

Note that a related problem exists in JSON: While JSON objects are not tied to a fixed format, JSON has no native mechanism for sums. Instead, we would typically use explicit tags to encode sums:

{ "drugName": "Paracetamol",
  "dosageKind": "tablet",
  "morning": 1,
  "midday": 0,
  "evening": 2
}

Again, this example emphasizes the need to separate between the data model in the software and encodings in a database or serialization formats, and use an anti-corruption layer between them as necessary.

Sums and Products and the Open/Closed Principle

Consider writing functions or methods that operate on a sum of products, such as one that formats the dosage for human readers. These functions usually have to branch on the specific case of dosage. In Java, there are fundamentally two ways of doing this branching. The object-oriented way uses polymorphic method dispatch to execute different code for different kind of dosages.

public sealed interface Dosage {
    String format();

    record Tablet(int morning, int midday, int evening) implements Dosage {
        @Override
        public String format() {
            return morning + "-" + midday + "-" + evening;
        }

    }
    record Infusion(double speed, int duration) implements Dosage {
        @Override
        public String format() {
            return speed + "ml/min for " + duration + "h";
        }

    }
}

However, modern Java also offers a more functional way via pattern matching. (Pattern matching in Java is greatly influence by functional languages with algebraic data types, see JEP 394, JEP 440, JEP 441, JEP 455, JEP 456).

public class Main {
    static String formatDosage(Dosage d) {
        return switch (d) {
            case Tablet(int morning, int midday, int evening) ->
                    morning + "-" + midday + "-" + evening;
            case Infusion(double speed, int duration) ->
                    speed + "ml/min for " + duration + "h";
            // Java compiler checks that we cover all cases.
        };
    }

    static String formatMedication(Medication m) {
        return m.drugName() + ": " + formatDosage(m.dosage());
    }
}

The functional way has the advantage that the complete logic for formatting is in one place, so it is easy to reason about its behavior. Further, new operations (e.g. serializing/deserializing) can be added without touching the existing classes and interfaces. The disadvantage is that adding new kinds of dosages is somewhat painful because it requires extending all relevant switch-expressions with the new alternative. (Readers familiar with the visitor pattern might notice that this design pattern has similar tradeoffs: adding new operations is easy, adding new alternatives is hard.)

The object-oriented way via polymorphic method dispatch has advantages and disadvantages reversed. It is easy to add new kinds of dosages, one merely has to add a new class implementing Dosage without touching the existing code. But adding new operations is painful because it requires a new method in the Dosage interface, with changes to all classes implementing it.

The familiar Open/Closed Principle states that software - in the face of new requirements - should ideally only require extension, not modification. Code written in what we called the functional way (or with the visitor pattern) enables openness for new operations, whereas the object-oriented way enables openness for new alternatives.

The designers of functional languages and those who recently brought record/data classes and pattern matching into object-oriented languages felt that the tradeoffs of the functional way are worth considering. This is particularly the case with combinator models, a subject for another post.

Of course, it would be nice if both adding more cases and more functions would equally adhere to the open/closed principle. This is a problem in language design known as the expression problem.

The formatting code in Python can also be written via pattern matching. The static type checker pyright checks statically that the match covers all possible cases.

def format(m: Medication) -> str:
    return f'{m.drugName}: {formatDosage(m.dosage)}'

def formatDosage(d: Dosage) -> str:
    match d:
        case TabletDosage():
            return f'{d.morning}-{d.midday}-{d.evening}'
        case InfusionDosage():
            return f'{d.speed} ml/min for {d.duration}h'

Sums and Products in Various Languages

To illustrate programming with sums and products, we‘ve implemented representations for medication dosages along with the associated formatting function/method in various languages. For brevity, we list only the functionality for dosages, omitting the surrounding medication record. The full code is available.

Kotlin

Kotlin offers sealed interfaces and „data classes“ corresponding to Java‘s records. Kotlin does not offer pattern matching, but its flow-sensitive type system allows convenient access to the attributes of a summand. The compiler statically checks that a when covers all possible cases.

sealed interface Dosage {
    fun format(): String =
        when(this) {
            is Tablet ->
                "$morning-$midday-$evening"
            is Infusion ->
                speed.toString() + "ml/min for " + duration + "h"
        }
    data class Tablet(val morning: Int, val midday: Int, val evening: Int) : Dosage {}

    data class Infusion(val speed: Double, val duration: Int) : Dosage {}
}

C#

In C#, we use records to encode products. For sums, we have to resort to inheritance.

public record Dosage {
    public record Tablet(int morning, int midday, int evening) : Dosage();
    public record Infusion(double speed, int duration) : Dosage();

    public string format() {
        return this switch {
            Tablet t => t.morning + "-" + t.midday + "-" + t.evening,
            Infusion i => i.speed + "ml/min for " + i.duration + "h",
            _ => throw new ApplicationException("unexpected dosage: " + this)
        };
    }

    // private constructor can prevent derived cases from being defined elsewhere
    private Dosage() {}
}

The compiler cannot check that Tablet and Infusion are the only possible subtypes of Dosage, so the switch statement in format requires a default case _. The official proposal for adding union types to C# would allow us to omit the default case.

Racket/Teaching Languages

The Racket system has many languages. The code here is written in the DeinProgramm teaching languages. These have records for products, allow declaring sums as „mixed data“, and support pattern matching. There is no static checking that the match covers all possible cases.

#lang deinprogramm/sdp
(define-record tablet
  make-tablet
  (tablet-morning natural)
  (tablet-midday natural)
  (tablet-evening natural))

(define-record infusion
  make-infusion
  (infusion-speed rational)
  (infusion-duration natural))

(define dosage
  (signature (mixed tablet infusion)))

(: format-dosage (dosage -> string))
(define format-dosage
  (lambda (dosage)
    (match dosage
      ((make-tablet morning midday evening)
       (string-append (number->string morning) "-"
                      (number->string midday) "-"
                      (number->string evening)))
      ((make-infusion speed duration)
       (string-append
        (number->string speed) "ml/min for "
        (number->string duration) "h")))))

Clojure

Clojure offers records for products. Sums do not need to be explicitly declared. There is no static checking that the cond covers all possible cases.

(defrecord Tablet [morning midday evening])
(defrecord Infusion [speed duration])

(defn format-dosage
  [dosage]
  (cond
    (instance? Tablet dosage)
      (str (:morning dosage) "-" (:midday dosage) "-" (:evening dosage))
    (instance? Infusion dosage)
      (str (:speed dosage) "ml/min for " (:duration dosage) "h")))

Scala

Scala, a strongly typed language, has direct support for algebraic data types, called enumerations. The following is Scala 3 code. The compiler statically checks that a match covers all possible cases.

enum Dosage {
  case Tablet(morning: Int, midday: Int, evening: Int)
  case Infusion(speed: Double, duration: Int)

  def format =
    this match {
      case Tablet(morning, midday, evening) =>
        morning + midday + evening
      case Infusion(speed, duration) =>
        speed + "ml/min for " + duration + "h"
    }
}

F#

F# is another strongly typed language with algebraic data types and pattern matching. The compiler statically checks that a match covers all possible cases.

type Dosage
  = Tablet of int * int * int
  | Infusion of double * double

let formatDosage(dosage: Dosage): string =
	match dosage with
	| Tablet (morning, midday, evening) ->
	  string morning + "-" + string midday + "-" + string evening
	| Infusion (speed, duration) ->
	  string speed + "ml/min for " + string duration + "h"

Swift

Swift was inspired by strongly typed functional languages. It offers algebraic data types in the form of „enums“ as well as pattern matching. The compiler statically checks that a switch covers all possible cases.

enum Dosage {
    case Tablet(Int, Int, Int)
    case Infusion(Double, Int)
}

extension Dosage {
    func format() -> String {
        return switch self {
        case let .Tablet(morning, midday, evening):
            morning.formatted() + "-" + midday.formatted() + "-" + evening.formatted()
        case let .Infusion(speed, duration):
            speed.formatted() + "ml/min for " + duration.formatted() + "h"
        }
    }
}

Rust

Rust - being in many ways inspired by Haskell - has direct support for both algebraic data types and pattern matching. The compiler statically checks that match covers all possible cases.

enum Dosage {
    Tablet { morning: i32, midday: i32, evening: i32 },
    Infusion { speed: f32, duration: i32 }
}

fn format_dosage(dosage: Dosage) -> String {
    match dosage {
        Dosage::Tablet { morning, midday, evening } =>
            format!("{morning}-{midday}-{evening}"),
        Dosage::Infusion { speed, duration } =>
            format!("{speed} ml/min for {duration}h")
    }
}

Typescript

Typescript‘s type system has „undiscriminated unions“ via the | operator. It‘s up to the programmer to include a tag in the participants of a union to distinguish them. In the following example, the compiler can check that the switch covers all possible cases.

type Dosage = {
    kind: "tablet",
    morning: number,
    midday: number,
    evening: number
} | {
    kind: "infusion",
    speed: number,
    duration: number
}

function formatDosage(dosage: Dosage) {
    let d: string;
    switch (dosage.kind) {
        case "tablet":
            d = m.dosage.morning + "-" + m.dosage.midday + "-" + m.dosage.evening
            break
        case "infusion":
            d = m.dosage.speed + " ml/min for" + m.dosage.duration + "h"
            break
    }
    return m.drugName + ": " + d
}

Terminology

Why are these constructs called sums and products? One simple illustration uses the number of values a sum or product type has. Consider the following Java enumerations:

enum T2 {
    A, B
}
enum T3 {
    X, Y, Z
}

(Of course, a Java enum is also a limited form of sum type.)

T2 has two values and T3 has three. Here‘s a product of these two types:

record P(T2 t2, T3 t3) {}

This type has six values - the product of 2 and 3. Now consider a sum:

sealed interface S{}
record RT2(T2 t2) implements S {}
record RT3(T3 t3) implements S {}

This has 2+3=5 values. So sums correspond to sums of numbers and products to products of numbers.

Another way to look at these two constructs would be from a set-theoretic perspective: products are basically cartesian products and sums are set unions. As the programming-language constructs for sums in Haskell or Java ensure that the participants in a sum are distinguishable from each other, they are also called disjoint or discriminated unions.

Discussion

Sums and products are important building blocks of data models, allowing architects to create ergonomic, powerful software, durable architectures, and maintainable code. Despite the fundamental role these concepts play, it‘s unfortunate that universally accepted terms for sums and products are still lacking within the programming and architecture community.

For an extensive introduction to systematic data modeling with sums and products (using design recipes), check out Felleisen and colleagues‘ classical book How to Design Programs and the German book Schreibe Dein Programm!, both freely available online.

Sums and products are also covered in the iSAQB Advanced curriculi on Functional Architecture (FUNAR) and Domain-Specific Languages (DSL).

A Library for Configurations

Marcus Crestani — Mon, 06 Nov 2023 00:00:00 UTC

Software must be configurable to be flexible. A configuration defines parameters and settings for software. Usually, the settings are stored in a configuration file that the software reads. But how do we ensure that a configuration is complete and valid? That all aspects that need to be configured are actually configured? That there are sensible defaults for values not explicitly configured? And that the values entered in the configuration are actually sensible values?

To avoid answering these questions anew for each project, we have developed a library for configurations for Clojure and ClojureScript that we have been using successfully for many years - and present in this article.

Configuration

Following typical Clojure style, we represent configurations as nested key-value maps.¹ Such configuration maps consist of settings and sections. Here‘s a simple example of a configuration for an application with a web server:

{:log-level :info
 :webserver {:host "0.0.0.0"
             :port 80}}

The configuration contains the :log-level setting for configuring the application‘s minimum logging level, which is set to the value :info. It also contains a section :webserver for configuring the web server‘s properties. The :webserver section is itself a configuration map with settings :host for the listen host (here the web server is accessible from outside by specifying "0.0.0.0") and the listen port, namely the standard HTTP port 80.

You can already sense a possible pitfall with configurations: An application typically expects very specific configured values, which it then interprets and responds to accordingly. In our example, the application assumes that the value for the logging level is actually a keyword like :info and not, say, a string "info" or "INFO". And the web server port should be a number 80 and not a string "80" and so on.

And programmers can specify exactly such constraints with our library and then check configurations against these constraints, ensuring that the application receives a valid and understandable configuration.

Configuration Schema

To do this, the application defines a configuration schema that it expects. Such a configuration schema is called a schema in our library. Now let‘s define the settings, sections, and then the schema for the above example configuration in order. In the code examples, we assume that the namespace active.clojure.config is imported as config.

Let‘s start with the definition of the logging level setting:

(def log-level-setting
  (config/setting
   :log-level
   "Minimal log level, defaults to :error."
   (config/one-of-range #{:trace :debug :info :warn :error :fatal}
                        :error)))

The call to config/setting expects the setting‘s keyword as its first argument, here it‘s :log-level. Next is a mandatory string documenting the setting – clear descriptions are very helpful for reference, showing users what they can configure and when the configuration causes errors. As the third argument, we define the valid value range for the setting, just called a range in our library. The value range for the log level is a one-of-range, meaning a valid value is one of the specified keywords :trace, :debug, :info, :warn, :error, or :fatal. Additionally, the range allows specifying a default if the setting is not explicitly configured. Here it‘s :error – this fact is also helpfully described in the description text.

Following the same pattern, we now write the settings for the web server. First for the host:

(def webserver-host-setting
  (config/setting
   :host
   "The address the webserver listens on, defaults to 0.0.0.0."
   (config/default-string-range "0.0.0.0")))

The setting for the web server host should be a string, so we use a string range. In this case, the range additionally allows specifying a default "0.0.0.0", so we use default-string-range to define this. Our library already contains a variety of ranges for commonly used value ranges, and it‘s easy to define custom ranges for more specialized value ranges. We‘ll see another built-in range with the next configuration setting.

The definition of the configuration setting for the port looks like this:

(def webserver-port-setting
  (config/setting
   :port
   "The port the webserver listens on, defaults to 3000."
   (config/integer-between-range 0 65535 3000)))

The configuration expects the port as an integer, so we use an integer range. However, in TCP networks there are only a limited number of ports, namely from 0 to 65535, so we restrict the possible value range to that with integer-between-range. The third argument to integer-between-range is the default if the setting was not explicitly made. Here our application uses port 3000.

These two settings make up the web server section – we bundle them together in a schema with an appropriate description:

(def webserver-schema
 (config/schema
  "Configuration settings for the web server"
  webserver-host-setting
  webserver-port-setting))

And with this schema, we can now define the web server section:

(def webserver-section
  (config/section
   :webserver
   webserver-schema))

No description is needed here, since the description is already in the schema.

Now we have everything we need for our complete configuration schema:

(def schema
 (config/schema
  "Configuration for our application"
  log-level-setting
  webserver-section))

Validation and Normalization

With this, we can now check configurations against our schema for validity:

(config/normalize&check-config-object
  schema
  {:log-level :info
   :webserver {:host "0.0.0.0"
               :port 80}})

The call returns the configuration that was passed in, meaning that the passed configuration is valid and complete.

Incomplete configurations are completed with defaults: The call

(config/normalize&check-config-object
  schema
  {:webserver {:host "0.0.0.0"}})

returns the completed configuration with the defined defaults:

{:log-level :error
 :webserver {:host "0.0.0.0"
             :port 3000}}

Finally, an erroneous configuration

(config/normalize&check-config-object
  schema
  {:webserver {:port "80"}})

returns a data structure called RangeError that describes the error in the configuration via a path to the erroneous configuration setting, the incorrect value, and the actually expected value range. This gives users the ability to understand and fix the problem. The representation of the above RangeError printed with str looks like this:

"Range error at path [:webserver :port]: value \"80\" is not in range integer between 0 and 65535"

Accessing Configuration Settings

Now we could read the settings from the nested configuration map using the settings keywords. But this isn‘t a particularly robust approach, since the Clojure compiler won‘t even notice possible typos in the keywords – in case of doubt, such access to a non-existent key simply returns nil. And that in turn could confuse or disrupt our application – which we want to prevent through robust configuration.

Therefore, we don‘t want to access the configuration map directly with keywords, but rather use safer mechanisms available in our library:

We don‘t handle the map directly, but work with a special data type called Configuration.
We use the settings and sections bound to variables to access the values – this way the compiler notices possible typos at compile time.
We use functions for access that ensure a validated configuration.

We‘ll show what this looks like in practice shortly. First, let‘s bind our example configuration to a name for our next experiments and explanations:

(def c
  (config/make-configuration
    schema
    {:log-level :info
     :webserver {:host "0.0.0.0"
                 :port 80}}))

From this configuration, we can now read the values for the settings using the access function:

(config/access c log-level-setting)

returns :info.

And reading from nested configurations works like this:

(config/access c webserver-port-setting webserver-section)

This returns 80.

It‘s also possible to extract an entire section and read values from it:

(let [webserver-config (config/section-subconfig c webserver-section)]
  (config/access webserver-config webserver-port-setting))

This enables configuration of different application components without too much coupling.

Lenses on Configuration Settings

Lenses play a major role in our daily work – and therefore also in this blog – because they have great practical utility. Therefore, our configuration library naturally knows how to work with lenses, for example for accessing settings. The code

(let [log-level-lens (config/access-lens log-level-setting)]
  (log-level-lens c))

returns :info, just like the direct access above. Noteworthy is that we can construct the lens without the configuration – a very elegant way to write selectors.

The same works for nested configurations:

(let [webserver-port-lens (config/access-lens webserver-port-setting
                                              webserver-section)]
  (webserver-port-lens c))

This returns 80 as expected.

Projection of Configuration Settings

Avoiding or at least reducing coupling is the most important mantra for good software. One method for decoupling is to use different data structures for different areas of the application, even if they are very similar.²

Therefore, it‘s worthwhile in software to separate settings from configuration – for example by introducing a data structure that represents the settings but is not the Configuration data structure. Our library makes this easy through an interplay of records and projection lenses.

We can define the settings data structure we want to use in our application as a record:

(define-record-type Settings
  {:projection-lens settings-projection-lens}
  make-settings
  settings?
  [log-level settings-log-level?
   webserver-host settings-webserver-host
   webserver-port settings-webserver-port])

And define a projection lens between this record and the configuration:

(def configuration->settings
  (settings-projection-lens
    (config/access-lens log-level-setting)
    (config/access-lens webserver-host-setting webserver-section)
    (config/access-lens webserver-port-setting webserver-section)))

With this compact definition, we can now translate a configuration into settings³:

(configuration->settings c)

Profiles

Another useful feature is support for profiles. There are often variants of a configuration due to a particular environment or deployment. Our library helps with this too. For example, we can configure a profile for test environments that adjusts some aspects compared to the production environment. To do this, we add a profile named :test under :profiles in our example configuration. The complete configuration looks like this:

{:log-level :info
 :webserver {:host "0.0.0.0"
             :port 80}
 :profiles
 {:test {:log-level :debug}}}

In the test profile, we‘ve configured that we want to see debug logs in the test environment.

When reading the configuration, we can then „mix in“ the settings for this profile; the profile-specific settings then override the general settings. To do this, we call normalize&check-config-object additionally with a list of profiles to consider:

(config/normalize&check-config-object
  schema
  [:test]
  {:log-level :info
   :webserver {:host "0.0.0.0"
               :port 80}
   :profiles
   {:test {:log-level :debug}}})

This then returns the completed configuration for the test environment with logging level :debug:

{:log-level :debug
 :webserver {:host "0.0.0.0"
             :port 80}}

Conclusion

An important foundation for flexible applications is robust configuration. The presented library enables this robust handling of configuration. The library has been successfully used in production in all our Clojure projects for many years.

Key-value maps, and thus our configurations as well, are valid EDN format, a common data transfer format in the Clojure ecosystem; saving and loading configuration files in this format works with Clojure‘s built-in spit, slurp, and read-string – we won‘t go into detail about this in this article. ↩
But as software grows and requirements change, similarities often don‘t persist. ↩
This even works in the other direction, since projection lenses are bidirectional. For example, if the application allows users to change settings and then wants to persist them as configuration in the configuration file. ↩

Monads in Kotlin

Michael Sperber — Mon, 22 May 2023 00:00:00 UTC

This post is part of the series on functional software architecture in Kotlin. The first one covered functional validation, and this part is about monads. In Kotlin, these are particularly useful when describing domain workflows that should be separated from the technical logic for executing these workflows - specifically using small domain-specific languages (DSLs).

This episode is about how monads actually work. Kotlin has - like many functional languages - special syntax for this, even though you won‘t find it under the „M-word“ in the documentation. It‘s hidden behind the suspend keyword.

The `Option` Type

Let‘s try one of the simplest monads, namely the Option type, also occasionally known as Maybe in typed functional languages, or Optional in Java. It‘s responsible for when operations sometimes return a result, but sometimes don‘t.

We‘ll build the type ourselves so we can show how to define the appropriate monad based on a type definition. We build it as a small type hierarchy with an interface and the obligatory two cases - Some, if a value exists, and None, if not:

sealed interface Option<out A> {
    companion object {
        fun <A> some(value: A): Option<A> = Some(value)
        fun <A> none(): Option<A> = None

        fun <A> Option<A>.get(): A =
            when (this) {
                is None -> throw AssertionError("found None where Some was expected")
                is Some -> value
            }
    }
}

object None : Option<Nothing>
data class Some<out A>(val value: A) : Option<A>

(For those who haven‘t seen out and Nothing in Kotlin yet: Don‘t worry, they clarify the interaction between generics and type hierarchies, but have no special conceptual significance.)

To make Option a monad, we need a bind method with the following signature:

sealed interface Option<out A> {
    fun <B> bind(next: (A) -> Option<B>): Option<B>
}

We implement this interface as follows for None and Some:

object None : Option<Nothing> {
    override fun <B> bind(next: (Nothing) -> Option<B>): Option<B> = None
}
data class Some<out A>(val value: A) : Option<A> {
    override fun <B> bind(next: (A) -> Option<B>): Option<B> = next(this.value)
}

We can now use this bind to implement a kind of „Poor Man‘s Exception Handling,“ that is, sequence many operations, each of which can fail (i.e., return None), and yield a final result only if everything succeeds. Unfortunately, this isn‘t syntactically pleasant, not even for die-hard parenthesis fans like myself:

some(5).bind { o1 ->
some(7).bind { o2 ->
  some(o1 + o2) } }

This is effectively Callback Hell, and even if we throw in a nitro cold brew flatrate, we‘re unlikely to lure OO programmers from their lair to routinely program monadically.

Haskell, Scala, and F#, have syntactic built-in that lets us write programs as a sequence of „statements,“ which the compiler then translates into calls to bind and related functions. In Haskell, for example, the code would look like this:

do o1 <- Just 5
   o2 <- Just 7
   return (o1+o2)

(In Clojure, for example, we can define such syntactic sugar ourselves.)

Coroutines and Continuations

Kotlin also has syntax as elegant as Haskell‘s, but it‘s hidden in a mechanism called coroutines. The documentation makes readers believe that coroutines are primarily about „asynchronous“ or concurrent programming, but behind it is a general mechanism that‘s activated for functions whose definition is marked with the suspend keyword. This puts the compiler into a different mode, which then performs a so-called CPS transformation.

„CPS“ stands for Continuation-Passing Style and is a particular way of writing functions. Normally we write programs „nested,“ where the result of a function call is returned.

f(g(h(x)))

With CPS, nothing ever goes back: When a function is finished, it doesn‘t „return“ a result, but instead calls a function that continues - namely the continuation. In CPS, the nested function call above looks like this:

h(x) { g(it) { gr -> f(it) { ... } } }

The program is thus linearized - the function calls appear in the order they happen at runtime. Additionally, each intermediate result gets a name. This, along with the curly braces, already has a certain similarity to the calls to bind above.

So why does the Kotlin compiler perform a CPS transformation? This is actually motivated by „asynchronous programming,“ where the goal is to avoid blocking a JVM thread because it‘s waiting for I/O, for example. Why avoid this? Because JVM threads consume a lot of memory and a JVM is only allowed to create a limited number of them. It would be better if a thread could do something else when a computation would block and then reactivate it when the I/O is done. For this, the program would need to remember the place in the code where it continues.

Normally, the JVM knows where to continue after a method call by consulting the stack, an implicit thing that a Java program cannot directly manipulate. In CPS, however, „how to continue after a method call“ is a continuation, i.e., an object that can be stored and used to reactivate a computation.

To program asynchronously with this, the program needs access to that continuation object, and there‘s a function called suspendCoroutine that calls a passed function with the current continuation, for which there‘s a special class in Kotlin. The continuation in turn has a resume method that continues execution until it calls suspendCoroutine again. We can use this to sneak in a call to bind and thus turn a „completely normal“ suspend function into a monadic program.

This is quite fiddly, but it works - here is the code in our small library that we wrote for this purpose. It provides a MonadDSL object with some helpful functions, whose definitions we omit here precisely because of this fiddliness.

Monad Syntax as a Kotlin DSL

We can use the abstractions from our library to first turn an Option<A> value into a computation that we can use in a suspend function:

sealed interface Option<out A> {
    suspend fun susp(): A = MonadDSL.susp<Option<A>, A>(this::bind)
}

We now use this method to define a small DSL. „DSLs“ in Kotlin typically means that we provide a set of functions locally available to a lambda expression by defining a function type „with receiver type“. Here‘s what that looks like:

fun <A> optionally (block: suspend OptionDSL.() -> A): Option<A> =
	MonadDSL.effect(OptionDSL(), block)

With the type OptionDSL.() -> A, OptionDSL is such a „receiver type“ and it ensures that the Kotlin compiler automatically makes every lambda expression passed to optionally inherit from the OptionDSL class, which is then able to use its functions. Additionally, they automatically get the suspend tag and are thus CPS-transformed by the Kotlin compiler.

The OptionDSL object contains only a single operation:

object OptionDSL {
    suspend fun  <A> pure(result: A): A = MonadDSL.pure(result) { Some(it) }
}

Larger monads naturally have more operations. This allows us to write programs like this:

optionally {
  val o1 = pure(5)
  val o2 = pure(7)
  pure(o1 + o2)
}

The result is Some(12) - and the Option monad is complete, including beautiful syntax!

A future post will cover larger monads and what role they can play in software architecture.

Data Conversion with Lenses

Marcus Crestani — Tue, 28 Feb 2023 00:00:00 UTC

Lenses are an important concept in functional programming with great practical utility. Several years ago, we already introduced functional lenses here on the blog. Since then, we have continuously expanded our use of lenses in our daily work. Today we want to show how we use lenses as bidirectional transformations and how they help us convert between data representations.

We implement the example for this article with our comprehensive and freely available Clojure library called Active Clojure, which we use in all our Clojure projects. This library includes implementations of lenses and Records, the latter of which is for representing compound data in Clojure. The combination of lenses and records is particularly helpful.

Data Structures

Every program processes data. Structuring this data and finding suitable data models is one of the most important tasks of good and successful software development. We want to turn data structures into types in our programs; for compound data, records are suitable, see also an older blog post. As an example, let‘s consider a data definition for a bookmark manager of a web browser that links descriptions with URLs. A URL consists of a protocol, a hostname, an optional port number, and a path:

(define-record-type URL
  make-url
  url?
  [protocol url-protocol
   host url-host
   port url-port
   path url-path])

The above record definition provides a constructor called make-url that expects as arguments values for the four fields protocol, host, port, and path. Additionally, the record definition provides a predicate url? that checks whether an argument is of type URL, and for the four fields the selectors url-protocol, url-host, url-port, and url-path. Thus the expression

(def url-1 (make-url "https" "funktionale-programmierung.de" 443 "/"))

binds an instance of a URL for the internet address of our blog to the name url-1. Access to the host field is possible with the selector

(url-host url-1)

and returns

"funktionale-programmierung.de"

Records and Lenses

Actually, the selectors are not just simple functions that return the values of the fields; they are lenses that focus on these values. From lenses you can read out the focused value with yank, so you can also write this for the above access:

(lens/yank url-1 url-host)

Since every lens performs an implicit lens/yank when applied to an argument (namely the data structure), we will also subsequently omit lens/yank for all other lenses and use the more compact, clearer selector notation.

With lenses you can also modify the focused values in data structures with shove:

(lens/shove url-1 url-host "active-group.de")

The above expression returns a new URL record that points to the homepage of Active Group (shove does not mutate the old URL record and therefore has no side effect).

Analogous to yank, there is also a more compact notation for shove. For this, you apply a lens to two arguments: the data structure and the new value:

(url-host url-1 "active-group.de")

is synonymous with the above shove expression¹.

Bookmarks

Now we continue with the data definition of our bookmark manager. A bookmark is a link that consists of a description and a URL:

(define-record-type Link
  make-link
  link?
  [description link-description
   url link-url])

Moreover, our bookmark manager consists of a list of all our bookmarks:

(define-record-type Bookmarks
  make-bookmarks
  bookmarks?
  [bookmarks bookmarks-bookmarks])

With this we can define the following bookmarks:

(make-bookmarks
 [(make-link "Largest German-language FP blog"
             (make-url "https" "funktionale-programmierung.de" 443 "/"))
  (make-link "Functional software development"
             (make-url "https" "active-group.de" 443 "/"))])

Data Conversion

Data conversion is the transformation of structured data into another data format – usually into a simpler serializable format that is better suited for transmission or storage.

For our bookmark manager, Extensible Data Notation (EDN for short) is suitable as a data format for storing or transmitting information. This could look like this:

{:bookmarks [{:description "Largest German-language FP blog"
              :url         {:protocol "https"
                            :host     "funktionale-programmierung.de"
                            :port     443
                            :path     "/"}}
             {:description "Functional software development"
              :url         {:protocol "https"
                            :host     "active-group.de"
                            :port     443
                            :path     "/"}}]}

Data conversion is needed very frequently when writing programs. With today‘s widespread client-server architecture, constant communication and data transmission between client and server is necessary, so as a software developer, one is constantly dealing with converting data representations for daily work. This is often tedious, means a lot of typing work, and is error-prone.

Projection Lenses

The implementation of data conversions becomes much easier with projection lenses. Projection lenses are lenses that connect two lenses with each other and can thus link values in different data formats. A simple example helps understand this idea:

Let‘s first take two simple EDN data structures:

a map

{:a 23
 :b 42}

and a vector

[23 42]

The keywords :a and :b are simultaneously lenses that focus on the respective values in the example map. To obtain lenses on the elements of the vector, we can use the index in the vector, so here (lens/at-index 0) for the first and (lens/at-index 1) for the second element.

A projection lens that projects the value of the map for the key :a to the first position of the vector and the value for :b to the second position of the vector connects the corresponding lenses and looks like this:

(def map->vector
  (lens/projection [] {(lens/at-index 0) :a
                       (lens/at-index 1) :b}))

We pass the mapping of which lenses map to each other as a map. Note that projection lenses have a direction and need a neutral element for the target data structure. Here the direction of yank is from the map to the vector, hence the empty vector is the neutral element. By reversing the mappings and adjusting the neutral element, we can change the direction in the lens/projection call. Alternatively, there is a lens invert that reverses a lens. A reversed lens also needs a neutral element for the target data format:

(def vector->map (lens/invert map->vector {}))

Converting the map to the vector then corresponds to a yank of this lens on the map:

(map->vector {:a 23 :b 42})

returns [23 42].

And shove on an empty map and the vector or a yank on the inverted lens converts back to the map representation {:a 23 :b 42}:

(map->vector {} [23 42])
(vector->map [23 42])

Projection Lenses for Records

Records can automatically generate appropriate projection lenses to make usage even easier. This works with the optional argument :projection-lens in a record definition, which specifies the name of the binding for the generated projection lens:

(define-record-type URL
  {:projection-lens into-url-projection-lens}
  make-url
  url?
  [protocol url-protocol
   host url-host
   port url-port
   path url-path])

The projection lens into-url-projection-lens already knows the lenses of the associated record fields. We can connect these fields with the lenses of our EDN data structure by passing them in the correct order – matching the order of the fields and the order in the constructor call – to into-url-projection-lens:

(def edn->url (into-url-projection-lens :protocol :host :port :path))

We can now use this lens for conversion between record representation and EDN representation; the record projection lenses are defined with the record as target. Thus, in our case they start from the EDN data structure, so a yank on the lens

(edn->url {:protocol "https"
           :host     "funktionale-programmierung.de"
           :port     443
           :path     "/"})

returns the record data representation:

(make-url "https" "funktionale-programmierung.de" 443 "/")

and shove into the empty map converts the record back to EDN format:

(edn->url {} (make-url "https" "funktionale-programmierung.de" 443 "/"))

All of this naturally also works for nested data structures. Here are the definitions still missing for the rest of the bookmark example, first for a bookmark:

(define-record-type Link
  {:projection-lens into-link-projection-lens}
  make-link
  link?
  [description link-description
   url link-url])

(def edn->link
  (into-link-projection-lens :description
                             (lens/>> :url edn->url)))

We achieve the correct conversion of nested data structures through composing lenses using the lens/>> operator. We link :url with the projection lens for URLs defined above, edn->url.

Now we still need the list of bookmarks:

(define-record-type Bookmarks
  {:projection-lens into-bookmarks-projection-lens}
  make-bookmarks
  bookmarks?
  [bookmarks bookmarks-bookmarks])

(def edn->bookmarks
  (into-bookmarks-projection-lens (lens/>> :bookmarks (lens/mapl edn->link))))

Here we achieve the correct conversion of the nested data structures with another combination of lenses: First we compose with lens/>> and then we use lens/mapl, a lens combinator that applies the passed lens to all elements of a list.²

With this we have all the components needed and can use the elegantly defined lens edn->bookmarks to convert our complete data definition between records and EDN.

Mixed Data

Our solution is extensible: As a new feature of our bookmark manager, we want to introduce folders for better organization. We want to allow folders everywhere we previously allowed individual bookmarks. We now have mixed data: elements of a bookmark list can be both links and folders.

A folder consists of a name and a list of bookmarks:

(define-record-type Folder
  {:projection-lens into-folder-projection-lens}
  make-folder
  folder?
  [name folder-name
   bookmarks folder-bookmarks])

Even within a folder, all bookmarks can be either links or folders again, so we are dealing with a mixed and mutually recursive data definition here. For dealing with mixed data, there is a built-in lens combinator union-vector. And for the mutual recursion, we don‘t need to handle it differently in the conversion to EDN than we have to do in Clojure anyway. In our bookmark->edn lens, we reference a value edn->folder that is still to be defined, so we must declare the name edn->folder beforehand and delay the evaluation of the value with the lens defer, passing edn->folder as a Clojure variable object:

(declare edn->folder)

(def bookmark->edn
  (lens/union-vector [link? (lens/invert edn->link)]
                     [folder? (lens/invert (lens/defer #'edn->folder))]))

As arguments, lens/union-vector receives a list of alternatives, where an alternative is a pair of a predicate and a lens that matches the data format to which the predicate applies. The specified lens thus uses a projection lens link->edn for links when the predicate link? matches; and a projection lens folder->edn when it matches the predicate folder?. Since we are projecting starting from the records, we must invert the projection lenses:

(def link->edn
  (lens/invert edn->link))

(def folder->edn
  (lens/invert edn->folder))

The lens edn->link remains unchanged from above, but the lens edn->folder is still missing. For this we need another conversion, since, in a folder, bookmarks can again consist of links and folders. Therefore we first define a projection:

(def edn->bookmarks-with-folders
  (into-bookmarks-projection-lens (lens/>> :bookmarks
                                           (lens/mapl (lens/invert bookmark->edn)))))

And with this we have all the parts needed to use the projection lens of the Folder record for the definition of edn->folder:

(def edn->folder
  (into-folder-projection-lens :name
                               (lens/>> :bookmarks
                                        (lens/mapl edn->bookmarks-with-folders))))

We can now try this out: converting to EDN and back to the record data representation returns the original value:

(edn->bookmarks-with-folders
 (bookmarks-with-folders->edn
  (make-bookmarks [(make-folder
                    "Blogs"
                    (make-bookmarks
                     [(make-link "Largest German-language FP blog"
                                 (make-url "https" "funktionale-programmierung.de"
                                           443 "/"))]))
                   (make-link "Functional software development"
                              (make-url "https" "active-group.de" 443 "/"))])))

We implemented the extension without having to change previously written code. Great!

Conclusion

Projection lenses are a suitable abstraction for conversions between different data formats. Their use is simple, not very error-prone, extensible and easily maintainable, thus facilitating software development.

Looking Beyond

This way of combining lenses to transform data is closely related to Pickler Combinators by Andrew J. Kennedy.

The presented record projection lenses are also excellent for interaction with the lenses in our configuration library active.clojure.config. We will present this in another blog post.

The more compact notation does not work with keywords used as lenses. ↩
Through the ability to use arbitrary lens combinators, the actual conversion has no limits. Another useful lens combinator in practice is lens/xmap. This, for example, makes it possible to transform values into other representations and back, for example transform between different date and time formats. ↩

Functional Validation in Kotlin

Michael Sperber — Thu, 19 Jan 2023 00:00:00 UTC

This post is the beginning of a series about functional software architecture in Kotlin. Its material originally comes from a collaboration between Active Group and Blume2000, where we consulted them on the development of their web shop. I wrote this post together with Benedikt Stemmildt, who was CTO at Blume2000 at the time.

This post is about data validation. We want to ensure that objects in our program are „valid,“ meaning they satisfy arbitrary consistency criteria without which our software won‘t function.

My colleague Marco Schneider already presented abstractions for this in Haskell in an earlier post.

The same ideas can also be transferred to Kotlin — with some compromises. In this post, we‘ll tackle the topic anew, specifically how we can help an object-oriented perspective using functional techniques. So it‘s not necessary to read the Haskell post. (We still warmly recommend the post, as it particularly describes the concept of Applicative, which is impractical in Kotlin.) However, basic Kotlin knowledge is assumed.

Premise

Martin Fowler nicely describes the validation problem in his blog post Martin Fowler: Replacing Throwing Exceptions with Notification in Validations. He quotes the following Java code fragment from a class whose attributes date and numberOfSeats must have certain properties for the class methods to work:

public void check() {
   if (date == null) throw new IllegalArgumentException("date is missing");
   LocalDate parsedDate;
   try {
     parsedDate = LocalDate.parse(date);
   }
   catch (DateTimeParseException e) {
     throw new IllegalArgumentException("Invalid format for date", e);
   }
   if (parsedDate.isBefore(LocalDate.now())) throw new IllegalArgumentException("date cannot be before today");
   if (numberOfSeats == null) throw new IllegalArgumentException("number of seats cannot be null");
   if (numberOfSeats < 1) throw new IllegalArgumentException("number of seats must be positive");
}

This code is quick to write and easy to read, but has at least two problems:

It throws an exception when a problem occurs. Stylistically, however, it makes sense to use exceptions primarily for situations where something unexpected and unavoidable happens in the software‘s environment („file not found“). Here, though, we‘re dealing with situations that are unpleasant but expected.
More importantly, the method checks for multiple possible problems but exits at the first one, so it can only report one problem.

In the Java ecosystem, frameworks have therefore been established that work without exceptions and collect and report all identified problems, such as the Hibernate Validator. Here‘s a code example for validation with such a validator, from the Kotlin code of the web shop at Blume2000:

data class Position(
    @field:Min(1, message = "COUNT_ZERO") var count: Int,
    @field:Valid var price: Price,
    @field:Valid var product: Product
)

The things with @ are annotations that the Kotlin compiler transfers into the object code for the Position class, where they are read by the validator at runtime. For example, the @field:Min annotation means that the value of the count field must be at least 1. The @field:Valid annotation refers to the validation annotations that are in the Price or Product class, respectively.

The validator only becomes active when explicitly called on a Position object:

val position: Position = ...
val factory = Validation.buildDefaultValidatorFactory()
val validator = factory.getValidator()
val violations: Set<ConstraintViolation<Position>> = validator.validate(position)

So the basic procedure is as follows:

first create an object and
check whether it‘s valid

Functional programmers‘ hackles rise at this approach. Why create an invalid object at all? We‘re reminded of the scene in Alien where the crew lets the alien into the spaceship for examination afterwards. (It doesn‘t end well.)

The „validator method“ also creates some very concrete problems:

The validator annotations effectively constitute a small programming language that you first have to learn.
The annotations couple the code to the validator framework.
Valid and invalid objects have the same type, so the type system can‘t help distinguish them.

Validation in Functional Programming

Accordingly, in functional programming we pursue a different approach to validation. The prominent functional programmer Yaron Minsky (CTO at Jane Street) coined the following saying:

„Make illegal states unrepresentable.“

Minsky doesn‘t speak directly about validation but about using the type system: In strongly typed languages, we should design our types so that only consistent, „legal“ data can be created at all.

Enforcing this with the type system alone isn‘t always possible, for example with numbers that must come from a specific range. Often we need, for instance, a type for natural (i.e., non-negative integer) numbers, but many languages only have int as a type for integers. To ensure that it‘s a non-negative number, we therefore have to do something at runtime.

But even at runtime we can follow Minsky‘s credo and prevent „invalid“ objects from being created in the first place. To implement this for the Position type from above, instead of the „official“ constructor function Position:invoke, we write a special, validating constructor function (or „factory method“ in OO speak) something like this:

class Position {
  companion object {
    fun of(count: Int, price: Price, product: Product) =
        if (count >= 1)
          ...
        else
          ...
  }

If we don‘t want to throw an exception in the else case—as in Martin Fowler‘s cautionary example — we need to accommodate the validation result in the return type of of. For this purpose, we use the Kotlin FP library Arrow, which contains all sorts of useful functional abstractions—particularly the type Validated. (Unfortunately, Arrow‘s documentation is somewhat elliptical at the time of this article‘s writing.) Validated is defined as follows:

sealed class Validated<out E, out A>
data class Valid<out A>(val value: A) : Validated<Nothing, A>
data class Invalid<out E>(val value: E) : Validated<E, Nothing>

Thus a Validated value can either encapsulate a valid value of type A with the Valid class, or with Invalid, a description of a validation error of type E. With this we can complete of as follows:

fun of(count: Int, price: Price, product: Product)
    : Validated<List<ValidationErrorDescription>, Position> =
    if (count >= 1)
      Valid(Position(count, price, product))
    else
      Invalid(listOf(MinViolation(price, 1)))

The type ValidationErrorDescription isn‘t listed here — it contains classes with descriptions of possible validation errors. We use a whole list of them here because multiple validation errors can occur when constructing a single complex object.

Pragmatics

In practice, however, you would make two more changes:

First, Arrow has defined two „extension functions“ that attach valid and invalid methods to every class, which call the constructors:

fun <A> A.valid(): Validated<Nothing, A> = Valid(this)
fun <E> E.invalid(): Validated<E, Nothing> = Invalid(this)

With these, the above code would idiomatically look like this:

fun of(count: Int, price: Price, product: Product)
    : Validated<List<ValidationErrorDescription>, Position> =
    if (count >= 1)
      Position(count, price, product).valid()
    else
      listOf(MinViolation(price, 1)).invalid()

(I personally find this confusing.)

Furthermore, most uses of Validated contain a list in the error case, more precisely a non-empty list. Arrow provides a convenience type alias for this:

public typealias ValidatedNel<E, A> = Validated<NonEmptyList<E>, A>

(The type NonEmptyList is also included with Arrow.)

Unfortunately, the extension functions valid and invalid don‘t work unchanged for ValidatedNel; there are extra functions validNel and invalidNel for that. With these, the constructor of finally looks like this:

fun of(count: Int, price: Price, product: Product)
    : ValidatedNel<ValidationErrorDescription, Position> =
    if (count >= 1)
      Position(count, price, product).validNel()
    else
      MinViolation(price, 1).invalidNel()

So much for constructing Validated. To process a Validated object, a when is sufficient:

when (Position.of(count, price, product)) {
  is Valid -> ...
  is Invalid -> ...
}

(Validated also has a fold method, but it‘s not necessarily conducive to readability.)

Anyone who looks deeper into Arrow will see that—following the Haskell package validation—it‘s possible to use any semigroup for type E. However, this is too cumbersome for practice in Kotlin since the semigroup isn‘t automatically inferred. Besides, ValidatedNel is usually sufficient anyway.

To write tests where you already know the objects are valid, we typically use a convenience function like this:

class ValidationException(message: String, val error: Any) : IllegalArgumentException(message)

@Throws(ValidationException::class)
fun <E, A> Validated<E, A>.get(): A = 
  this.valueOr { throw ValidationException("Validated expected to be valid", it as Any) }

The valueOr method is built into Arrow and either delivers the valid value in a Validated or calls the passed function, which in this case throws an exception.

With this we can simply write code such as this in tests:

val position: Position = Position.of(...).get()

Composition

Perhaps you‘ve wondered why of doesn‘t take ValidatedNel<..., Price> and ValidatedNel<..., Product> as arguments. After all, Price and Product probably also need to be validated. However, this isn‘t necessary since we follow the credo of not creating invalid Price and Product objects in the first place — Price and Product are thus implicitly already validated.

Nevertheless, we need to figure out how to combine the validations for Price and Product with that of Position. (In Haskell this happens with the help of the Applicative abstraction, which would be too cumbersome to use in Kotlin.) So let‘s imagine there are also factory methods for Price and Product:

class Price {
  companion object {
    fun of(...): ValidatedNel<ValidationErrorDescription, Price> = ...
  }
}

class Product {
  companion object {
    fun of(...): ValidatedNel<ValidationErrorDescription, Product> = ...
  }
}

To validate a Price and a Product object in parallel such that any errors are combined, Arrow provides a series of extension functions called zip. Each of these accepts a certain number of Validated values and applies a function to the results, if possible. For example, here‘s the three-argument zip:

public inline fun <E, A, B, C, Z> ValidatedNel<E, A>.zip
   (b: ValidatedNel<E, B>, c: ValidatedNel<E, C>, f: (A, B, C) -> Z)
   : ValidatedNel<E, Z>

This takes some getting used to because the first argument comes before the .zip and all others in the parentheses after it. We could call it like this:

Price.of(...).zip(Product.of(...))
  { price, product -> Product(count, price, product) }

However, this only works if we call the constructor of Product directly. But we naturally want to use Product.of, which itself returns a Validated. Unfortunately, Arrow doesn‘t offer a really practical abstraction here. It‘s easiest with our own helper function that uses zip to construct a pair from the two validated intermediate results:

fun <Z, A, B> validate(
  a: ValidatedNel<Z, A>,
  b: ValidatedNel<Z, B>
): ValidatedNel<Z, Pair<A, B>> = a.zip(b) { validA, validB ->
  Pair(validA, validB)
}

This works correspondingly for tuples of higher arity.

We can then process the „validated pair“ from validate with the extension function andThen, which is included with Arrow:

public fun <E, A, B> Validated<E, A>.andThen(f: (A) -> Validated<E, B>): Validated<E, B> =
  when (this) {
    is Validated.Valid -> f(value)
    is Validated.Invalid -> this
  }

Here‘s how it works:

validate(Price.of(...), Product.of(...))
  .andThen { (price, product) -> Product.of(count, price, product) }

(The validate function has the additional advantage of being symmetric, unlike zip.)

Note that andThen — just as in Haskell — has the same signature as a monadic bind or flatMap, but it doesn‘t form a monad: andThen doesn‘t accumulate errors.

Conclusion

Functional validation only requires a simple data type and a few methods on it and works without a DSL or annotations. Since „valid“ and „invalid“ are expressed by different objects, you could also call it „object-oriented validation.“ It‘s a good idea in any case.

In Kotlin, Arrow brings the right abstractions; only documentation and convenience are still somewhat lacking.

Event Processing with Riemann

Marcus Crestani — Wed, 28 Sep 2022 00:00:00 UTC

Riemann is a stream processing system that is excellent for collecting and processing events and logs from servers and systems. We successfully use Riemann in production in very large IT systems as a core component for log and metric processing and for monitoring.

We use Riemann to process events and forward them to long-term storage systems like Elasticsearch, where these events can be conveniently searched through user interfaces like Kibana. Additionally, we use Riemann to write metrics to time-series databases like InfluxDB; these time series can then be visualized through user interfaces like Grafana or can also be used to alert on errors and problems.

Today we‘ll show with an application example how to extract metrics from logs using Riemann.

Riemann

Riemann is written in Clojure and can also be configured and programmed in Clojure. In fact, the configuration of a Riemann instance is a Clojure program that controls event processing using a powerful and efficient stream processing language.

Riemann enables sending events through streams that can filter, modify, combine, aggregate, and project these events. There are dozens of built-in streams and it‘s very easy to write custom streams, since a stream is simply a function that accepts an event.

Events

Events are represented in Riemann as maps, i.e., as a collection of key-value pairs, like for example this event that represents a log line of a request to a web server:

(def request-event
   {:timestamp   1663577804126
    :method      "GET"
    :request     "/index.html"
    :requestor   "192.168.1.23"
    :transaction "uid-82a9dda829"
    :service     "webserver-request"
    :host        "192.168.1.1"
    :time        1})

We bind the event to the name request-event so we can use it for tests. Riemann defines some special fields like :host and :service that appear in all events; otherwise there are no restrictions on which fields and values an event can contain.

Streams

As already mentioned, a stream in Riemann is a function that accepts an event. In Riemann, „stream“ doesn‘t mean a sequence of data elements, which is what this term usually means in most contexts. Perhaps „event processor“ would be a better name for what a stream actually is in Riemann: a callback that accepts events and in turn passes these events on to other event processor callbacks (and has no return value). However, we‘ll continue to use the term stream, as this is the common designation in Riemann.

A simple stream is, for example, a where stream, which is built into Riemann:

(where <condition> <stream>)

A where stream is itself an event processor that accepts an event, checks if the specified condition is true, and then calls the event processor <stream> with the event.

With this we build a stream that separates log events from metrics: log events should be forwarded to an Elasticsearch server for long-term storage and metrics to an InfluxDB time-series database.

(streams
  (where
    (metric nil)
    elasticsearch-stream
    (else
      influxdb-stream)))

The where stream checks if flowing events match the predicate (metric nil). The predicate (metric nil) means that an event doesn‘t contain a :metric field, which in our example means that an event is not a metric. (Riemann‘s event taxonomy assumes that metrics are distinguished from other event types by the existence of the :metric field.) In this case, the event flows on to an elasticsearch-stream that finally sends it to the Elasticsearch server. In the else branch there are only metrics left, which Riemann then forwards to InfluxDB via the influxdb-stream.

The definitions of elasticsearch-stream and influxdb-stream look like this:

(def elasticsearch-stream
  (tap :elasticsearch-stream elasticsearch))

(def influxdb-stream
  (tap :influxdb-stream influxdb))

The tap in each of these definitions is a construct for intercepting events for test cases. We‘ll make use of this shortly.

The actual definitions of elasticsearch and influxdb would exceed the scope of this blog article, so we‘ll simply rely on wishful thinking and assume that they do what they‘re supposed to: forward the incoming events to Elasticsearch and InfluxDB respectively. For those particularly interested, we use the implementation from our Riemann library active-riemann for these streams.

Tests

Unit tests are indispensable; we also need to test our stream definition. Riemann brings support for unit tests; we can run tests with riemann test on the command line. But we first need to write the tests. The first test ensures that events that are not metrics only end up in Elasticsearch:

(deftest event-test
  (let [actual (inject! [request-event])]
    (is (= [request-event] (:elasticsearch-stream actual)))
    (is (empty? (:influxdb-stream actual)))))

The inject! function only exists to feed the streams in the context of unit tests. It sends a list of events into the defined streams. Here there‘s only one event in the list, the request-event defined above. The return value of inject! contains all events that have passed by the program points we marked with tap. Since we‘re listening to our stream to Elasticsearch with :elasticsearch-stream, we expect that we‘ll find exactly this one event there, because our stream definition has routed it there. And since we didn‘t inject a metric, we expect that no events have passed by :influxdb-stream, so this list must be empty, i.e., match the empty? predicate.

For the next test we want to inject a metric, so we create an example metric. For this we first build a helper function in which we abstract the appearance of our metric. The metric has a :label field for the label and a :metric field for the metric value, which we can pass to the helper function:

(defn make-webserver-transaction-metric
  [duration]
  {:label  "webserver-transaction-duration-milliseconds"
   :metric duration})

Then we generate our example metric with the value 116:

(def example-metric (make-webserver-transaction-metric 116))

For the next test case we inject this example metric:

(deftest metric-test
  (let [actual (inject! [example-metric])]
    (is (empty? (:elasticsearch-stream actual)))
    (is (= [example-metric] (:influxdb-stream actual)))))

Here we expect that no event has passed by :elasticsearch-stream, but the example metric appears at :influxdb-stream. All tests pass.

Web Server Logs

For our application example we assume that logs are generated on an imaginary web server. The requests make it into our Riemann system in the format described above.¹ In addition to the requests, the web server also logs the delivery of the response. An event for this looks like this:

(def reply-event
  {:timestamp   1663577804242
   :transaction "uid-82a9dda829"
   :service     "webserver-reply"
   :host        "192.168.1.1"})

The web server links request and response with a unique transaction ID in the :transaction field.

Determining Transaction Time

For our use case we‘re interested in how long the web server takes to answer a request. We want to measure and record this metric in order to monitor the web server‘s performance. This means we need to determine the elapsed time between the events of the request and response of a transaction and write it to our InfluxDB for storage, later visualization, or possibly also for alerts. If we subtract the timestamps from the :timestamp fields from each other, we get the transaction time in milliseconds. For our example this results in a transaction time of 116 milliseconds.

The crucial part is that we only calculate the distance of the associated log entries, so we need to find the log entries that have the same transaction ID. Specifically, this means we need to remember the requests until we see the matching response, i.e., the response with the same transaction ID as the request. Remembering something means state. Riemann brings support for storing state in the form of an index.

Index

The index is a storage for events built into Riemann. Riemann stores the most recently seen event for each tuple of :host and :service. We can query these events from the index, and the events in the index also have an expiration date: the :ttl field sets the „time-to-live“ for an event in the index in seconds. And we can specify at what intervals expired events should be deleted. Riemann not only deletes these events, but also notifies about these expired events—we can also use this information meaningfully for our goal of determining transaction time.

We initialize our index as follows:

(def default-ttl 60)

(def index
  (do
    (periodically-expire (/ default-ttl 10))
    (index)))

(def index-stream
  (sdo
   (tap :index identity)
   (default :ttl default-ttl index)))

First we define our default expiration time default-ttl as 60 seconds; we bind a new index to the name index and define with periodically-expire that Riemann should check for expired events every six seconds. Then we create a new index with the call to the Riemann function (index). We wrap the new index in a stream with the name index-stream. More precisely, our index-stream is an sdo composition of two streams: sdo is do for streams, so events flow into all streams in the body. Here an incoming event passes through the streams (tap :index identity) and (default :ttl default-ttl index):

The stream (tap :index identity) is only there so we can listen to our index stream for our unit tests.
The stream (default :ttl default-ttl index) provides each event that doesn‘t yet have a :ttl field with a default value for the :ttl field, namely with the value of our defined default lifetime. Subsequently, these events are then passed to the index to store them there.

Writing to the Index

We need a stream that writes a request event to the index. We call the stream store-requests-stream:

(def store-requests-stream
  (smap (fn [request]
          (clojure.set/rename-keys request {:transaction :service}))
        index-stream))

The stream uses an smap stream to apply the specified function to each event — i.e., to each request — and then forward the result into the index-stream. The function that smap applies transforms a request so that the index can store it correctly: as mentioned above, the index stores one event per tuple of :host and :service. In order to really store all events, we simply rename the :transaction field to :service in the request and thus have the unique transaction ID as part of the key in the index. The original value of :service doesn‘t interest us further. An indexed event looks like this, for example:

(def request-event
  {:timestamp   1663577804126
   :method      "GET"
   :request     "/index.html"
   :requestor   "192.168.1.23"
   :service     "uid-82a9dda829"
   :host        "192.168.1.1"
   :time        1})

Reading from the Index

When response events arrive, we need to read associated requests from the index. We call this stream lookup-requests-and-calculate-metric-stream; the stream expects response events:

(def lookup-requests-and-calculate-metric-stream
  (smap (fn [reply]
          (when-let [request (riemann.index/lookup index
                                                   (:host reply)
                                                   (:transaction reply))]
            (riemann.index/delete index request)
            (make-webserver-transaction-metric (- (:timestamp reply)
                                                  (:timestamp request)))))
        reinject))

This stream also uses an smap stream to call a function on each response event and then reinject the result back into our streams via the reinject stream, i.e., push it back into the front of our streams.

In the body of this function several things happen: riemann.index/lookup tries to find a possibly stored request based on the :host and :transaction fields of the response. If this succeeds, the function removes the found event from the index so that it can‘t expire later — after all, it‘s already been processed. The result of this function is a metric that we create with make-webserver-transaction-metric and which calculates the difference of the timestamps between request and response as its value. If a matching request is not in the index, for example because we started our program after this request would have arrived, the function returns nil. This then has the intended effect that this nil is not reinjected, since smap doesn‘t pass the value nil on to the streams.

Keeping the Index Clean

In order for requests to not accumulate for which the web server — for whatever reason — never sent out a response, we‘ve given each request in the index a maximum lifetime. When Riemann removes such expired requests from the index, it reinjects information about this into the streams, in the form of the request, but additionally marked with a :state field with the value expired. With the predicate (state "expired") Riemann can filter these events—and we can use this information.

In case of such a timeout we want to output a metric whose value should be very large, i.e., Double/MAX_VALUE in our number space. We call the metric timeout-metric:

(def timeout-metric
  (make-webserver-transaction-metric Double/MAX_VALUE))

We call the stream for this emit-timeout-metric-stream:

(def emit-timeout-metric-stream
  (smap (constantly timeout-metric) reinject))

The smap stream reinjects a metric with Double/MAX_VALUE into the streams for each incoming value, regardless of the type of value.

Don‘t Cross the Streams

With this we have all the components together; now we can supplement our original stream with a few switches to achieve our desired functionality:

(streams
 (where
  (metric nil)
  (sdo
   (where (not (state "expired")) elasticsearch-stream)
   (split
    (service "webserver-request") store-requests-stream
    (service "webserver-reply") lookup-replies-and-calculate-metric-stream
    (state "expired") emit-timeout-metric-stream))
  (else
   influxdb-stream)))

The case for metrics remains unchanged. The case for events that are not a metric becomes more complicated: the sdo marks that the events should be passed on to multiple streams. First is the stream that forwards to Elasticsearch. However, we don‘t want to see the information about the expired requests there, so we filter them out with (not (state "expired")). Otherwise all requests and all responses should end up in Elasticsearch.

The next stream implements the calculation of the transaction metrics. We split the event stream in a split stream into requests with the predicate (service "webserver-request"), into responses with the predicate (service "webserver-reply"), and into expired events with the predicate (state "expired"):

The requests end up in the index via the store-requests-stream defined above.
The responses lead to metrics via lookup-replies-and-calculate-metric-stream and the requests previously stored in the index.
The expired requests lead to metrics that mark the timeout.

„Why not?“ — „That would be bad!“

Two unit tests should secure our implementation. The first tests a complete transaction consisting of request and response:

(deftest transaction-test
  (let [actual (inject! [request-event reply-event])]
    (is (= [request-event reply-event]
           (:elasticsearch-stream actual)))
    (is (= [indexed-event]
           (:index actual)))
    (is (= [example-metric]
           (:influxdb-stream actual)))))

The test injects our example events for request request-event and response reply-event. Our expectations:

Both events are sent to Elasticsearch.
The request ends up in modified form as indexed event indexed-event in the index.
The calculated metric example-metric reaches InfluxDB.

The second test verifies the behavior in the timeout case:

(deftest transaction-expired-test
  (let [actual (do
                 (inject! [request-event])
                 (riemann.time.controlled/advance! (* 2 default-ttl))
                 (inject! [reply-event]))]
    (is (= [request-event reply-event]
           (:elasticsearch-stream actual)))
    (is (= [indexed-event]
           (:index actual)))
    (is (= [timeout-metric]
           (:influxdb-stream actual)))))

Here we first inject only the request request-event and then advance the clock for this test with riemann.time.controlled/advance! so far forward that the request in the index has definitely expired. Only then do we send the response reply-event, which then finds no matching request. Consequently, we see both request and response event in Elasticsearch, and also the indexed request event passed through the index. However, the only metric that made it to InfluxDB is the timeout metric timeout-metric. In addition to the behavior in the timeout case, this unit test even covers the behavior in the case where no request is present in the index for a response event.

With this we‘ve tested all branches of our streams and achieved our goal: we‘ve derived metrics from incoming events with Riemann.

Conclusion

Riemann is performant and efficient in execution - our systems running in production process several thousand events per second despite large and complicated stream logic.

And Riemann is elegant to program: with the extensive stream processing language, typical tasks in event processing like filtering, enriching, combining, aggregating, and projecting can be easily accomplished. And for all other tasks we can flexibly extend the stream processing language.

„Total Protonic Reversal!“

For example, Logstash is suitable for reading log files, processing them, and forwarding them to Riemann. ↩

Higher-Kinded Data for Configurations in Haskell

Felix Leitz — Fri, 29 Jul 2022 00:00:00 UTC

Many applications use configurations to customize their behavior at runtime. The parameters in these configurations can have default values that are used if nothing else is specified. Other values, such as passwords, have no default values and must therefore always be provided when starting the application.

In this article, we will go through several iterations to model configurations in our programs using higher-kinded data in Haskell.

Note: Accompanying source code can be found on Github.

First Attempt

Suppose we have a program that requires a password, the URL of a service, and its port as configuration. We could then model the configuration with the following data type.

data Config = Config
  { password :: String
  , serviceUrl :: String
  , servicePort :: Int
  }
  deriving (Show)

We assume that the parts of the configuration are read from environment variables at program startup. If an environment variable is not set, or if the given value cannot be interpreted, a default value should be used. Additionally, the password must always be provided at runtime and therefore has no default value.

The function that implements all of this could look like this:

getConfig :: IO Config
getConfig = do
  pw <- getPassword
  url <- fromMaybe "localhost" <$> getUrl
  port <- fromMaybe 8080 <$> getPort
  pure (Config pw url port)

getPassword :: IO String
getPassword = do
  mPassword <- lookupEnv "PASSWORD"
  pure (fromMaybe (error "Environment variable PASSWORD not set") mPassword)

getUrl :: IO (Maybe String)
getUrl = lookupEnv "SERVICE_URL"

getPort :: IO (Maybe Int)
getPort = do
  mPortStr <- lookupEnv "SERVICE_PORT"
  pure (readMaybe =<< mPortStr) 

We use the function:

lookupEnv :: String -> IO (Maybe String)

which returns the value of an environment variable if it is set.

We see three helper functions here that handle reading and interpreting the respective parts of the configuration. In the getConfig function, everything is assembled. This function constructs values of the Config data type and also merges of default values with the read values.

This approach has at least two problems:

The getConfig function mixes building the configuration with reading the individual parameters and combining them with default values. This means it‘s not immediately clear whether a field has a default value.
Other configuration types must write their own getConfig function.

Second Attempt

Another possibility would be to choose the following data type:

data Config' static dynamic = Config
  { password :: dynamic String
  , serviceUrl :: static String
  , servicePort :: static Int
  }

As we can see, two type parameters have been introduced compared to the first Config' data type. Both parameters are type functions with kind Type -> Type. By choosing these two type functions, we can achieve the following: dynamic values cannot be specified at compile-time, while static values can.

To implement the semantics behind static and dynamic, we use the two built-in data types Identity and Proxy. Identity a is a data type that contains a value of type a. This means that in all definitions of a configuration, these values must be present. Proxy a, on the other hand, does not contain a value of type a. Thus we can define a value of type Proxy a without having to provide a value of type a.

We define the following type aliases to make the assignment of static and dynamic clearer in different situations.

type DefaultConfig = Config' Identity Proxy
type PartialConfig = Config' Maybe Identity
type Config = Config' Identity Identity

PartialConfig here replaces static with the Maybe type constructor and dynamic with Identity. Thus, for partial configurations, static values can potentially be provided, while dynamic values must be provided. In a value of type Config, both static and dynamic values must be provided.

With this, we can now define a default configuration:

defaultConfig :: DefaultConfig
defaultConfig =
  Config
    { password = Proxy
    , serviceUrl = Identity "localhost"
    , servicePort = Identity 8080
    }

Although this definition only contains the values we know statically, calls to Proxy and Identity are still necessary here, which impairs readability.

The function that reads the environment variables at runtime now looks like this:

readInPartialConfig :: IO PartialConfig
readInPartialConfig = do
  password <- Identity <$> getPassword
  url <- getUrl
  port <- getPort
  pure (Config password url port)

Note here that both the URL and the port can be read, but do not necessarily have to be present. The password, on the other hand, must be loaded dynamically.

The last missing building block is the function that combines the default and partial configuration into the final Config. Values from the PartialConfig are preferred over those from the DefaultConfig.

combineConfig :: DefaultConfig -> PartialConfig -> Config
combineConfig
  (Config _defaultPasswordProxy defaultServiceURL defaultServicePort)
  (Config pw url port) =
    Config
      pw
      (maybe defaultServiceURL Identity url)
      (maybe defaultServicePort Identity port)

The getConfig function now simply uses the functions defined so far:

getConfig :: IO Config
getConfig = combineConfig defaultConfig <$> readInPartialConfig

With our new definition of the configuration data type, we were able to fix the first of the above problems: From the definition of Config', it is clearly recognizable which fields have default values and which do not, and there is a value defaultConfig that contains all these values.

Unfortunately, the code is not reusable in this form. For a new configuration, we still have to rewrite the data type and some functions.

Type Families to the Rescue

To make the code reusable, we define the type family HKD (Higher-Kinded Data). This is how we model the application of type functions to types:

type HKD :: (Type -> Type) -> Type -> Type
type family HKD f a where
  HKD Identity a = a
  HKD f a = f a

Through this definition, we save ourselves unnecessary calls to Identity later.

Our configuration type is unchanged except for the call to HKD in the field signatures. Also, for technical reasons, we need to have an instance of the Generic type class generated.

data Config' static dynamic = Config
  { password :: HKD dynamic String
  , serviceUrl :: HKD static String
  , servicePort :: HKD static Int
  }
  deriving (Generic)

The definition of defaultConfig is also almost the same, except for the missing calls to the Identity constructor:

defaultConfig :: DefaultConfig
defaultConfig =
  Config
    { password = Proxy
    , serviceUrl = "localhost"
    , servicePort = 8080
    }

Similarly for the partial configuration:

readInPartialConfig :: IO PartialConfig
readInPartialConfig = do
  password <- getPassword
  url <- getUrl
  port <- getPort
  pure (Config password url port)

Because we derived the Generic type class in the definition of Config', we can write a function genericApply with the following somewhat simplified type.

genericApply :: c Identity Proxy -> c Maybe Identity -> c Identity Identity

This function takes over the job of the combineConfig function. We can implement it using GHC.Generics without using the definition of Config'. Thus, genericApply can be moved to a library and does not have to be rewritten for each configuration type. The exact implementation goes too far here; it can be found in the Github repo linked above.

By replacing the type variable c with Config', we get the same type as for combineConfig. With this, we have everything needed to build the getConfig function:

getConfig :: IO Config
getConfig = do
  partialConfig <- readInPartialConfig
  pure (genericApply defaultConfig partialConfig)

With this iteration of Config', we finally did it. For a new configuration type, we only need to define the type, the default value, and the reading of dynamic values. We get combining for free. In particular, the HKD type family, the genericApply function, and the necessary helper functions can be moved to a library, since they are always the same regardless of the Config' definition.

With three more small type aliases, it‘s even possible to remove Proxy and Identity:

type Default c = c Identity Proxy
type Partial c = c Maybe Identity
type Complete c = c Identity Identity

dynamic :: Proxy a
dynamic = Proxy

These definitions can also be moved to the library.

Finally, the complete configuration definition looks like this:

data Config' static dynamic = Config
  { password :: HKD dynamic String
  , serviceUrl :: HKD static String
  , servicePort :: HKD static Int
  }
  deriving (Generic)

type DefaultConfig = Default Config'
type PartialConfig = Partial Config'
type Config = Complete Config'

defaultConfig :: DefaultConfig
defaultConfig =
  Config
    { password = dynamic
    , serviceUrl = "localhost"
    , servicePort = 8080
    }
  
readInPartialConfig :: IO Config
readInPartialConfig =do
  password <- getPassword
  url <- getUrl
  port <- getPort
  pure (Config password url port)

getConfig :: IO Config
getConfig = do
  partialConfig <- readInPartialConfig
  pure (genericApply defaultConfig partialConfig)

Conclusion

With the help of type functions and Higher-Kinded Data, we managed to separate the default values of our configuration from reading the dynamic values. We were able to make clear at the type level which values are known dynamically and which are known statically. Finally, we were able to extract functions so that only relevant parts need to be rewritten.

Functional Programming in Practice: Validation with Applicative Functors

Marco Schneider — Tue, 26 Apr 2022 00:00:00 UTC

Our first article in the series „Functional Programming in Practice“ on the topic of data validation with applicative functors.

Welcome to our first article in the series „Functional Programming in Practice“. In this series, we present concepts from functional programming using real examples and show how we actually apply them in practice every day.

Today we‘ll focus on data validation and show how applicative functors help us systematically collect and process inconsistencies in data.

Validating JSON

Web services like to communicate with each other using JSON objects that end up in our application from users and other services. It‘s no secret that the JSON format is limited by its narrow range of types. In practice, this means that complex data types are usually serialized as strings. Additionally, we often have specific requirements for the form of the concrete data: a "username" should not be empty, an "email" should contain at least an "@" symbol, and serialized values must match our expectations to be converted into internal data types.

For the following example, we‘ll work with this data analysis. A user consists of:

a name
an email address
a role

A role is one of the following:

a developer
an admin
a bug reporter

For the implementation, we‘ll use the Haskell programming language. It could look like this:

data UserRole = ADMIN | DEVELOPER | REPORTER

data User = { userName  :: String
            , userEmail :: String
            , userRole  :: UserRole }

-- Examples
marco = User "Marco" "marco.schneider@active-group.de" DEVELOPER
simon = User "Simon" "simon.haerer.at.active-group.de" DEVELOPER

The two examples in the code immediately show a problem: while we know that Simon‘s email address isn‘t valid, if we read it directly from a JSON object, nothing stops us from ignoring such an invariant („every email address contains an ‚@‘ symbol“). So we need to validate the data somehow before it penetrates further into the system.

We could, of course, write a constructor for User that catches such errors. We‘ll use Either a b = Left a | Right b for the result. Here:

An error is signaled with Left a
A success is signaled with Right b

type Error = String

makeUser :: String -> String -> Role -> Either Error User
makeUser name email role
  | '@' `elem` email = Right (User name email role)
  | otherwise        = Left  "not a valid email"

Great, problem solved. If everything goes smoothly, we get a Right User, otherwise a Left Error. But wait, what if the username is empty? A second attempt:

type Error = String

makeUser :: String -> String -> Role -> Either Error User
makeUser name email role
  | null name        = Left "not a nonempty string"
  | '@' `elem` email = Right (User name email role)
  | otherwise        = Left "not a valid email"

That‘s a bit better. But what if both the name is empty and the email doesn‘t contain a '@'? After all, we want to know all the errors, not just the first one. But that also means we need a different signature (with a list of errors for Left).

type Error = String
type Errors = [Error]

makeUser :: String -> String -> UserRole -> Either Errors User
makeUser name email role =
  let nameOk = not (null name)
      emailOk = '@' `elem` email
  in
    if nameOk && emailOk
    then Right (User name email role)
    else
      if nameOk && not emailOk
      then Left ["not a valid email"]
      else
        if not nameOk && emailOk
        then Left ["not a nonempty string"]
        else Left ["not a valid email" "not a nonempty string"]

Well, that‘s not really satisfying. Moreover, the number of cases we need to check in increasingly nested branches explodes. This won‘t work.

However - I think - we‘ve learned something about what we actually want:

We want to be able to perform each validation individually
We want to be able to combine multiple validations
We want to write a validation function that returns a validated result exactly when everything is fine or a list of all errors if something is wrong.

Validation Functions

To give this whole thing some structure, let‘s first define our own data type that corresponds to our validation results. Such a result is one of the following:

A successful validation (Ok <value>) of a value, or
a list of errors that occurred during validation (Fail Errors). Why a list? Imagine validating a web form with more than one form field. It would be annoying to only be told about one error at a time, only to see the next error after correcting it. We‘d rather have all errors at once.

type Errors = [String]

data Validation c = Ok c 
                  | Fail Errors

Our Validation represents the result of a validation. It‘s parameterized over the type of a candidate value c. This allows us to specify validations for values of any type.

Next, we‘ll write a few functions that represent our validations from above:

-- | Validate that `candidate` is not the empty string.
validateNonemptyString :: String -> Validation String
validateNonemptyString candidate
  | null candidate = Fail ["not a nonempty string"]
  | otherwise      = Ok candidate
  
-- | Validate that a string represents an email.
validateEmail :: String -> Validation String
validateEmail candidate
  | '@' `elem` candidate = Ok candidate
  | otherwise            = Fail ["not an email"]

-- | Validate that a `candidate` represents a `UserRole`.
validateUserRole :: String -> Validation UserRole
validateUserRole candidate
  | candidate == "ADMIN"     = Ok ADMIN
  | candidate == "DEVELOPER" = Ok DEVELOPER
  | candidate == "REPORTER"  = Ok REPORTER
  | otherwise                = Fail ["not a role"]

With this, we can already perform individual validations:

validateNonemptyString "ok"
-- Ok "ok" : Validation String
validateNonemptyString ""
-- Fail ["not a nonempty string"] : Validation String

validateEmail "marco.schneider@active-group.de" 
-- Ok "marco.schneider@active-group.de" : Validation String
validateEmail "marco.schneider.at.active-group.de" 
-- Fail ["not a valid email"] : Validation String

validateUserRole "ADMIN"
-- Ok ADMIN : Validation UserRole
validateUserRole "DEVELOPER"
-- Ok DEVELOPER : Validation UserRole
validateUserRole "unknown"
-- Fail ["not a valid role"]

Point one (perform each validation individually) is done now. Now, however, we want to combine multiple results with each other.

Combining Results

Let‘s first look at combining two error cases: since both errors consist of lists of error messages, we can concatenate them and retain all information. That sounds good!

combineValidations :: Validation a -> Validation a -> Validation a
combineValidations (Fail es) (Fail fs) = Fail (es ++ fs)

The Haskell compiler is, of course, still unsatisfied: what if on the left, right, or even in both positions of the function there‘s an Ok?

Our third requirement demands: Either a valid result or all errors. With just one Fail on the left side, we already know it can only result in a Fail, even if there‘s a success on the right. So in that case, we simply ignore the right side:

combineValidations :: Validation a -> Validation a -> Validation a
combineValidations (Fail es) (Fail fs) = Fail (es ++ fs)  -- from above
-- When there's an error on the left and a success on the right,
-- we're no longer interested in the success:
combineValidations (Fail es) _ = Fail es

Okay, we‘ve covered two of four cases - so far, so simple. To make the Haskell compiler happy, we still need two more cases:

Success on the left, error on the right
Success on the left, success on the right

This might sound strange at first, but: we functional programmers don‘t have an awful lot of tools at our disposal. We can feed functions with arguments and see what the result is (or in the case of Haskell, keep trying until the compiler gives its okay). So we don‘t have much choice at this point but to simply pretend that on the left there‘s an Ok whose value is a function that knows what to do with the value on the right. If we accept that for a moment, we can write a little helper function for it.

applyOkFunctionToValidation :: (a -> b) -> Validation a -> Validation b
-- Well, a failure remains a failure, 
applyOkFunctionToValidation _ (Fail es) = Fail es
-- However, we'd like to keep a success. So we take what's
-- wrapped in the success and apply `f` to it. If you look at
-- the type signature, there's not much else left (except for the trivial solution).
applyOkFunctionToValidation f (Ok c) = Ok (f c)

What does this say? Assuming we get a function from an Ok on the left side, we can use applyOkFunctionToValidation to apply the value on the right side to it. A Fail on the right side remains an error (so it propagates along the computation in a way) and remains unchanged. An Ok results in another Ok by applying the f wrapped in the left Ok to it. This way, we‘ve made one success out of two.

Now we could almost use applyOkFunctionToValidation in combineValidations, if it weren‘t for its type signature Validation c -> Validation c -> Validation c getting in the way - we need a Validation (a -> b) -> Validation a -> Validation b. But if you‘ve paid attention, you noticed: we haven‘t actually used the a in the signature yet (so far, we‘ve only been interested in the error, which has nothing to do with a). Thus, we can change the signature without any problem. With that, combineValidations is finished and can be used.

applyOkFunctionToValidation :: (a -> b) -> Validation a -> Validation b
applyOkFunctionToValidation _ (Fail es) = Fail es
applyOkFunctionToValidation f (Ok c) = Ok (f c)

combineValidations :: Validation (a -> b) -> Validation a -> Validation b
combineValidations (Fail es) (Fail fs) = Fail (es ++ fs)
combineValidations (Fail es) _ = Fail es
combineValidations (Ok f) v = applyOkFunctionToValidation f v

Isn‘t Something Missing?

Yes, kind of. Unfortunately, we can‘t just use our validation functions defined above as the first argument for combineValidations anymore - we want a function in the Ok. To get rid of that problem too, we‘ll take one last detour.

Imagine we have a validation on the left that, combined with a validation on the right, always returns the right result. Haskell already provides the function id : a -> a, whose result is always exactly its argument. If we wrap that in an Ok, it looks like this:

(Ok id) `combineValidations` validateNonemptyString ""
-- Fail ["not a nonempty string"]
(Ok id) `combineValidations` validateNonemptyString "Marco"
-- Ok "Marco"

Does that help us? A bit. It doesn‘t really matter which function is in the Ok. So we can quite generally turn any function into a validation:

makeValidation x = Ok x

makeValidation reverse `combineValidations` validateNonemptyString "Marco"
-- Ok "ocraM"

Doesn‘t help much, but it works. Although, it does help somehow, because quite inconspicuously a solution for points two and three from above has crept in here.

makeValidation User
  `combineValidations` (validateNonemptyString "Marco")
  `combineValidations` (validateEmail "marco.schneider@active-group.de")
  `combineValidations` (validateUserRole "DEVELOPER")
-- Ok (User {userName = "Marco", userEmail = "marco.schneider@active-group.de", userRole = ADMIN})

If you don‘t believe it, feel free to check the error cases:

-- Everything wrong
makeValidation User
  `combineValidations` (validateNonemptyString "")
  `combineValidations` (validateEmail "marco.schneider.at.active-group.de")
  `combineValidations` (validateUserRole "DEVELOPE")
-- Fail ["not a nonempty string","not an email","not a role"]

-- Partially wrong
makeValidation User
  `combineValidations` (validateNonemptyString "Marco")
  `combineValidations` (validateEmail "marco.schneider.at.active-group.de")
  `combineValidations` (validateUserRole "DEVELOPE")
-- Fail ["not an email","not a role"]

This probably looks magical at first: why does makeValidation User suddenly become a validation function when it only wraps the original constructor? There‘s no way around it, we need to briefly look at how to imagine the individual substitutions (important: this is only for illustration and doesn‘t really correspond to the execution machinery). I‘ll separate the representation of the execution steps with =>:

makeValidation User
  `combineValidations` (validateNonemptyString "")
  `combineValidations` (validateEmail "marco.schneider@active-group.de")
  `combineValidations` (validateUserRole "DEVELOPER")

=> Inserting the definition of `User`

(Ok (\name email role -> User name admin role))
  `combineValidations` (validateNonemptyString "Marco")
  `combineValidations` (validateEmail "marco.schneider@active-group.de")
  `combineValidations` (validateUserRole "DEVELOPER")

=> Result of the first right side

(Ok (\name email role -> User name admin role))
  `combineValidations` (Ok "Marco")
  `combineValidations` (validateEmail "marco.schneider@active-group.de")
  `combineValidations` (validateUserRole "DEVELOPER")

=> Inserting the first result into the left side

(Ok (\email role -> User "Marco" admin role))
  `combineValidations` (validateEmail "marco.schneider@active-group.de")
  `combineValidations` (validateUserRole "DEVELOPER")

=> Now this repeats for the remaining arguments

(Ok (\email role -> User "Marco" admin role))
  `combineValidations` (Ok "marco.schneider@active-group.de")
  `combineValidations` (validateUserRole "DEVELOPER")

=> 

(Ok (\role -> User "Marco" "marco.schneider@active.group" role))
  `combineValidations` (validateUserRole "DEVELOPER")

=>

(Ok (\role -> User "Marco" "marco.schneider@active.group" role))
  `combineValidations` (Ok DEVELOPER)

=>

(Ok (User "Marco" "marco.schneider@active.group" DEVELOPER))

Presto! Everything is fine. Analogously for the case when errors occur (we‘ll jump in where it gets interesting):

(Ok (\email role -> User "Marco" admin role))
  `combineValidations` (validateEmail "marco.schneider.at.active-group.de")
  `combineValidations` (validateUserRole "DEVELOPER")

=>

(Ok (\email role -> User "Marco" admin role))
  `combineValidations` (Fail ["not an email"])
  `combineValidations` (validateUserRole "DEVELOPER")

=> We remember from above: errors on the right consume `Ok`s on the left

(Fail ["not an email"])
  `combineValidations` (validateUserRole "dummy")

=> And once we have the error, we just collect the remaining errors together.

(Fail ["not an email"])
  `combineValidations` (Fail ["not a role"])

=>

(Fail ["not an email" "not a role"])

What helps us here is that in Haskell functions can be partially applied. That means a function with three arguments, when we apply one argument, results in a function with two arguments. In languages where this isn‘t the case (like Clojure, for example), we have to work a bit harder.

With that, we actually have everything we need to write our validation function:

validateUser :: String -> String -> String -> Validation User
validateUser name email role =
	makeValidation User
	  `combineValidations` validateNonemptyString name
	  `combineValidations` validateEmail email
	  `combineValidations` validateUserRole role

The official part is done, because our three criteria from above are fulfilled. We want to:

be able to perform each validation individually -> we write functions that have Validations as results
chain multiple validations together -> with combineValidations
write a validation function that returns a validated result when everything is fine or a list of all errors if something is wrong → with validateUser

The practical part is now complete. We‘ve looked at how we can write composable validation functions for arbitrary data types, how to link them together, and be sure that we really capture all errors.

However, if you‘re wondering what‘s so incredibly applicative about this and where the functor is hiding, you can continue enjoying yourself below.

Who or What is an Applicative Functor Here?

As functional programmers, we‘re always interested in finding more general properties and structures in our code (or designing it that way from the start). An example of our approach at home is the „Find-the-Functor-Game“™, because: where a functor is hiding, there might also be an applicative functor, there might also be a monad. Or generally: if you find a bit of structure, there‘s usually more.

Those who already know what makes a functor in programming have already spotted it above. In Haskell, you write it as a type class like this:

class Functor f where
  fmap :: (a -> b) -> f a -> f b

To avoid falling into metaphors about what a functor is, I‘d rather show where the functor was hiding above. If we look at the signature of fmap, it looks suspiciously like applyOkFunctionToValidation!

fmap                        :: (a -> b) ->          f a ->          f b
applyOkFunctionToValidation :: (a -> b) -> Validation a -> Validation b

So we could quite generally say that our Validation is a functor:

instance Functor Validation where
  fmap _ (Fail es) = Fail es
  fmap f (Ok c)    = Ok (f c)
  -- or more simply
  -- fmap = applyOkFunctionToValidation

Armed with this knowledge, we can make a first small change to the code from above.

combineValidations :: Validation (a -> b) -> Validation a -> Validation b
combineValidations (Fail es) (Fail fs) = Fail (es ++ fs)
combineValidations (Fail es) _         = Fail es
combineValidations (Ok f)    v         = fmap f v

That doesn‘t look like much, but it‘s already quite a bit. Our use of fmap at this point (and of course the implementation of Functor) signals to all readers that:

certain laws apply here (please don‘t miss the warning further below about laws)
all code from all libraries that know how to do something with functors is also usable for us.

Applicative Functor

It was somehow already clear that an applicative functor is hiding here - otherwise the title of the post would be quite a dud.

Let‘s first look again at the definition of the corresponding type class in Haskell:

class Functor f => Applicative f where
  pure  :: a -> f a
  (<*>) :: f (a -> b) -> f a -> f b

Both signatures might already look familiar to us:

pure           :: a ->          f a
makeValidation :: a -> Validation a

(<*>)              ::          f (a -> b) ->          f a ->          f b
combineValidations :: Validation (a -> b) -> Validation a -> Validation b

Then let‘s implement it:

instance Applicative Validation where
  pure  = makeValidation
  (<*>) = combineValidations

And finally:

validateUser :: String -> String -> String -> Validation User
validateUser name email role =
	pure User <*> validateNonemptyString name
	          <*> validateEmail email
	          <*> validateUserRole role

Nicely, we get (as already with the Functor) everything that‘s available for applicative functors for free. Just a small example: the <$> operator ((<$>) :: Functor f => (a -> b) -> f a -> f b, where (<$>) = fmap) makes the validation a little bit prettier still:

validateUser email name role =
  User <$> validateNonemptyString "Marco"
       <*> validateEmail "marco.schneider@active-group.de"
       <*> validateUserRole "DEVELOPER"

What Does All This Get Me?

You would right in wondering why we‘re going through all this effort. Why look for the functor? And even applicative functors? Since we‘re dealing here with the practical application of functional programming, I‘d like to pick out just one detail.

If you look at the type signature of (<*>) :: f (a -> b) -> f a -> f b (or even the definition), you‘ll notice: the individual arguments are independent of each other. That means, I‘m theoretically free in the expression

make <$> foo <*> bar <*> baz

to evaluate foo, bar, and baz in any order - or in any context (as long as the types „fit“)! This means, for example, that I could decide to write my implementation of <*> such that each argument is evaluated on its own thread and the result is called when all parallel computations are finished. This then naturally applies to every instance of Applicative that adheres to the laws - in this case, the associativity law.

pure (.) <*> u <*> v <*> w = u <*> (v <*> w)

It should hold that: the application of the composition pure (.) <*> u <*> v <*> w is equivalent to applying u to the result of v <*> w (or more simply put: composition of applicative values with pure (.) behaves like composition of functions with .). So it should not matter in which order the result is calculated.

Conclusion

We‘ve presented to you a variant of data validation in functional programming practice. To do this, we used an applicative functor and showed that with it we can very easily combine partial results and thus always know exactly what went wrong.

The accompanying source code for this article can be found on Github.

Important Postscript

One more note on the topic of type classes and signatures. Just because the signatures of functions look like they could fit a type class doesn‘t mean it will become a correct type class. Such a class usually consists of:

A set of operations (fmap or pure plus <*>)
A set of laws that must hold for values of the type under these operations

Unfortunately, Haskell doesn‘t give us the right means to ensure that the respective laws are also upheld by the instance. However, showing that our instances are correct would exceed the scope of this already long post.

Update July 9, 2022 Of course, applicative validation has nothing to do with Haskell or even static typing. You can see this, for example, in our Clojure library active-clojure. Here you‘ll find an implementation of applicative functors in the dynamically typed language Clojure.

Transducer: Composition, Abstraction, Performance

Marco Schneider — Thu, 22 Mar 2018 00:00:00 UTC

Transducer: Composition, Abstraction, Performance

Higher-order functions like map, fold, filter are indispensable in any functional program. With their flexibility, they are the tool of choice for operations on collections of all kinds. However, their scope of application is not limited to classic lists or vectors. In this article, we examine more fundamental properties of these operations and take a particular look at so-called transducers in the Clojure programming language.

Additionally, we will see how we can achieve not only very good reusability but also higher performance with the help of meaningful abstraction.

Everything is a `fold`

Those who have engaged with functional programming in the past may already be familiar with this statement. If you haven‘t had the opportunity yet (or want to refresh your memory), let‘s first look at a variant of the usual definitions of map and filter. (Note: All examples are written in Clojure, so from this point on we‘ll call fold by its Clojure name reduce):

(defn map
  "Takes a function f and applies it to every element of xs."
  [f xs]
  (if (empty? xs)
    xs
    (cons (f (first xs)) (map f (rest xs)))))

(defn filter
  "Takes a predicate pred and returns a list with all elements of xs that
  satisfy pred."
  [pred xs]
  (if (empty? xs)
    xs
    (if (pred (first xs))
      (cons (first xs) (filter pred (rest xs)))
      (filter pred (rest xs)))))

Many definitions can be found like this in various standard libraries. As the heading of this section already reveals, both functions can also be defined via fold (or rather reduce). The signature of reduce, expressed in Haskell notation, looks like this: (b -> a -> b) -> b -> [a] -> b (specialized for lists in this case).

(defn mapping
  "Takes a function f and returns a function. The returned function takes a
  collection acc and an element x and appends x applied to f to the end of
  acc."
  [f]
  (fn [acc x] (conj acc (f x))))

(defn map
  "Takes a function f and applies it to every element of xs."
  [f xs]
  (reduce (mapping f) [] xs))

(defn filtering
  "Takes a predicate pred and returns a function. The returned function takes
  a collection acc and an element x and appends x to the end of acc if it
  satisfies pred."
  [pred]
  (fn [acc x] (if (pred x) (conj acc x) acc)))

(defn filter
  "Takes a predicate pred and returns a list with all elements of xs that
  satisfy pred."
  [pred xs]
  (reduce (filtering pred) [] xs))

If our programs consisted exclusively of list processing, we could stop here, satisfied. However, something catches the eye: apart from the call to conj, the definitions of mapping and filtering tell us nothing about them working „only“ on collections!

From Collections to Processes

We have established that the connection to collections in our mapping and filtering functions exists only through conj. Now let‘s go one step further and see what happens when we abstract over conj. For this, it‘s helpful to think not of lists but of sequences of steps. mapping and filtering take on the role of „process modifications“: they receive a step and deliver a modified version of this step. So let‘s define a version of our functions parameterized over this „step“:

(defn mapping
  [f]
  (fn [step]
    (fn [acc x] (step acc (f x)))))

(defn map
  [f xs]
  (reduce ((mapping f) conj) [] xs))

(defn filtering
  [pred]
  (fn [step]
    (fn [acc x] (if (pred x) (step acc x) acc))))

(defn filter
  [pred xs]
  (reduce ((filtering pred) conj) [] acc))

Looking at the resulting definitions of mapping and filtering, we notice that the data structure on which they now operate is determined by the step parameter. This means we are no longer dependent on classic collections, but can (as we will see later) work with more or less arbitrary data structures and process them using mapping and filtering. However, one consideration for this step is still outstanding: while lists and vectors make it easy to think about a „beginning“ and an „end,“ which we use in map and filter, it‘s different with data structures like streams or signals. Instead of relying on the data structure, we take a different path:

Our transformers (mapping, filtering) should have a concept of „beginning“ and „end“ by themselves.
We expect from our step function not only that it is binary, but also that when called with no or one argument, it produces a „null“ value. Examples in Clojure would be (conj) => [], (conj [1]) => [1] or (+) => 0, (+ 1) => 1, etc.

We‘ll only address point 1 here. First, we‘ll propose an implementation of mapping and filtering. In the code example below, we make use of Clojure‘s syntax for „arity overloading.“ This means we can offer multiple implementations in one function for different numbers of arguments (more on this at Clojure - Functional Programming). Clojure will choose the corresponding implementation depending on the number of arguments passed.

(defn mapping
  [f]
  (fn [step]
    (fn
      ([] (step)) ; 1.
      ([acc] (step acc)) ; 2.
      ([acc x] (step acc (f x))) ; 3.
      )))

(defn filtering
  [pred]
  (fn [step]
    (fn
      ([] (step))
      ([acc] (step acc))
      ([acc x] (if (pred x)
        (step acc x)
        acc)))))

At first glance, this may look somewhat unintuitive, but it‘s quickly explained using the example of mapping. The numbering here corresponds to the numbers in the code example above.

This case covers the „beginning“ of a process. Nothing has been calculated yet and there is no „next“ calculation. In this case, we want a „neutral“ element from our step function with which we can kick off the calculation.
Here we signal the „end“ of a process. No more elements are coming, so we take the last step with the already available „result.“
Finally, there‘s our case that processes the current result and an element together.

These functions are now parameterized to such an extent that we can express somewhat arbitrary processes with them and, probably more importantly, execute (or compose) multiple such process modifications in sequence! What does this look like in reality?

Process Modifiers

In the following, we give two examples of how all this helps us in the real world and when we might prefer this approach to regular chaining of list operations.

In both examples, we deal with the following problem: The semester is over and the professor wants to know how good the average performance of their master‘s students was in the exercise sessions. Using Clojure-Spec, we define some sample data. The first four lines define the „shape“ of our data. For example, ::title is a value that satisfies the string?-predicate; in the fifth line, we specify ::exercise as a map with the keys defined above. The output of sample has been prettified here a bit; the actual result would probably look somewhat more obviously random. Those who want to learn more about Spec can do so in an older blog article (German language only).

(ns my.namespace
  (:require [clojure.spec.alpha :as s]
            [clojure.spec.gen.alpha :as sgen]))

(s/def ::title string?)
(s/def ::student string?)
(s/def ::points (set (range 31)))
(s/def ::degree #{::msc ::bsc})
(s/def ::exercise (s/keys :req [::title ::student ::points ::degree]))

;; `sample` takes a generator for our `spec` and returns some random data that
;; matches this spec.
(sgen/sample (s/gen ::exercise))
;; =>
;; [{::name "Marco"
;;   ::title "Exercise 1"
;;   ::points 29
;;   ::degree ::msc}
;;  {::name "Mathias"
;;   ::title "Exercise 1"
;;   ::points 28
;;   ::degree ::bsc}
;;  ...]

Back to our task. This could be solved with regular list functions, for example, like this:

(defn sum-of-msc
  [exercises]
  (reduce + 0 (->> exercises
                   (filter #(= ::msc (::degree %)))
                   (map ::handins))))

Applied to a list of exercises, it gives us the correct result. However, there is a problem: Each call to filter, map, and reduce computes a new list! With large amounts of data, this can, as we‘ll see shortly, certainly lead to problems.

Next, we implement the same functionality expressed via our newly defined operators.

The first step will be a slightly modified version of reduce named reduce-with. This works very similarly to reduce, except that in addition to the reduction function (here step), it expects a parameter xf, which represents a composition of our functions.

(defn reduce-with
  "Takes a function composed of process modifications xf, a step function, an 
  initial value and a sequence of operations. Applies xf with a step to every x
  in xs."
  [xf step init xs]
  (let [f (xf step)]
    (f (reduce f init xs))))

Inside the function, we call reduce as usual. However, the composition function xf is used as the reduction function, which is specialized for evaluation via the step parameter (itself a function) and bound to f. Then reduction occurs and the result is called once more with f to produce a final result. But what must xf look like? This function, quite analogously to sum-of-msc above, is also a composition of filtering and mapping. We call it xform here (a convention in the Clojure world).

;; Compose two process modificators into one.
(def xform
  (comp (filtering #(= ::msc (::degree %)))
        (mapping ::points)))

You may rightly wonder why the order of calls to filtering and mapping doesn‘t change, since regular composition (as with Clojures comp function) is evaluated „from right to left,“ but the threading in our sum-of-msc happens „from left to right“ (or top to bottom). At this point, the complete answer would exceed the scope. For the moment, let‘s just note that the composition of this type of function also evaluates „from right to left.“ You can easily find this out by completely reducing the composition of the functions by hand once.

(defn sum-of-msc-2
  [exercises]
  (reduce-with xform + 0 exercises))

This solution also delivers the correct result, also revealing a major speedup:

(let [exercises (sgen/sample (s/gen ::exercises) 1000)]  ; generate a random sample of 1000 exercises
  (= (time (sum-of-msc exercises))
     (time (sum-of-msc-2 exercises))))
;; => true
;; "Elapsed time: 407.378092 msecs"
;; "Elapsed time: 2.144632 msecs"

Upon closer inspection, the reason becomes obvious: Instead of computing a new list for each step, the processes are executed sequentially for each element. This saves us long intermediate results and gains us throughput.

Note: Obviously, the timing varies a lot by the actual kind of inputs we generate. Also, time is not really intended to be used as a measuring tool. However, the tendency that the transducer-version is vastly faster than the ‚standard‘ implementation still stands, even over repeated observations.

This concept is so practical that it is now part of the Clojure standard library under the name transducer. Many functions are already set up for use as transducers (including the well-known map, filter, mapcat, …). Our function reduce-with is defined there as transduce.

More Than Lists

Finally, the promised second example. The problem is the same, but this time we don‘t want to use lists but channels (via clojure.core.async). Here we write all values to a channel c and observe how the same xform process modification is applicable to these very channels. We also now use the built-in function transduce, which takes over the work of sum-of-msc-2.

(ns my.namespace
  (:require
    [clojure.core.async :as async]
    [clojure.spec.alpha :as s]
    [clojure.spec.gen.alpha :as sgen]))

(let [exercises (sgen/sample (s/gen ::exercises))
      ;; Create a channel using the xform defined above.
      c (async/chan 1 xform)]
      ;; Put all our elements onto the channel.
    (async/go (async/onto-chan c exercises))
    (let [chan-res (loop [;; Read one element from the channel.
                          n (async/<!! c)
                          res 0]
                     (if-not n
                       res
                       (recur (async/<!! c)  ; Read the next element from the channel.
                              (+ res n)  ; Accumulate the result.
                              )))
          list-res (transduce xform + 0 exercises)]
      (= chan-res list-res)))
;; => true

Here we first put all elements into the channel (onto-channel). Then we read step by step from this channel until no more elements are present in it (<!!). As we see from the result, our xform does exactly the same here as it did with lists. Since our xform doesn‘t care what data structure it operates on (as long as it provides the infrastructure mentioned above), it can be reused without any changes and without problems!

Conclusion

Conclusion Transducers are the natural continuation of widely used list functions. In this article, we engaged with consistent abstraction and learned the concept of transducer along the way. Not only do transducers give us a powerful description of data transformations that is purely based on function composition and can be used in many ways, but they also bring crucial performance improvements.“

Compound Data in Clojure

Michael Sperber — Mon, 27 Apr 2015 00:00:00 UTC

This post continues our Clojure introduction (Part 1, Part 2, Part 3). This time we‘re covering compound data.

In Part 2 of the Clojure introduction, we looked at built-in data structures. Today we‘ll cover the definition of new compound data types.

As an example, let‘s go to a computer store and put together a custom computer from individual components. To describe this configuration (for example, for an online shop), we‘ll design a representation as data. This process starts with a data definition:

; A computer consists of:
; - Processor
; - RAM capacity in Gbyte
; - Hard drive capacity in Gbyte

The data definition clearly states that a computer consists of several parts, so we need compound data. In Clojure, records are responsible for compound data, and we can define a record type for compound data using the defrecord form:

(defrecord Computer
    [processor ram hard-drive])

This definition creates a Java class named Computer, with fields processor, ram, and hard-drive. (Java programmers would call Computer a POJO class). Since Java classes typically start with a capital letter, we‘ll adopt this convention for record types.) This allows us to use the built-in Clojure new construct to create Computer objects:

(def gamer (new Computer "Cell" 4 1000)) ; Cell, 4 Gbyte RAM, 1000 Gbyte hard drive
(def workstation (new Computer "Xeon" 2 500)) ; Xeon, 2 Gbyte RAM, 500 Gbyte hard drive

We can abbreviate new Computer as Computer. (note the period):

(def workstation (Computer. "Xeon" 2 500))

However, it should be noted that Computer. is not a function, but a macro. If we try to use it as a standalone object, the following happens:

records> Computer.
CompilerException java.lang.ClassNotFoundException: Computer.

Fortunately, Clojure also provides a constructor function called ->Computer:

(def workstation (->Computer "Xeon" 2 500))

We can extract the individual parts of a record as follows:

records> (:processor gamer)
"Cell"
records> (:ram gamer)
4
records> (:hard-drive gamer)
1000

The keywords with the field names thus function as selectors - this is because in Clojure, every record also works as a map. (See our introduction to data types.)

That‘s actually enough to work with records in Clojure. However, compound data often also appears as cases in mixed data, for which we still need a predicate for a record type, i.e., a way to distinguish objects of the record definition from other objects.

To illustrate, we‘ll write a program to feed two types of animals: armadillos and parrots.

Here‘s a data definition for armadillos, a matching record definition, and an example armadillo:

; An armadillo has the following properties:
; - Weight (in g)
; - alive or dead
(defrecord Dillo
    [weight alive?])

(def d1 (->Dillo 55000 true)) ; 55 kg, alive

… and here‘s the same thing again for the parrot:

; A parrot has the following properties:
; - Weight in grams
; - Sentence it says
(defrecord Parrot
  [weight sentence])

(def p1 (->Parrot 10000 "The gardener did it.")) ; 10kg, Miss Marple

Feeding an armadillo increases its weight, at least if it‘s still alive:

(defn feed-dillo
  "Feed armadillo."
  [d]
  (if (:alive? d)
    (->Dillo (+ (:weight d) 500) true)
    d))

For a parrot, it always works:

(defn feed-parrot
  "Feed parrot."
  [p]
  (->Parrot (+ (:weight p) 50)
            (:sentence p)))

Now let‘s write a function that handles both armadillos and parrots. The data definition for this looks like:

; An animal is one of the following:
; - an armadillo
; - a parrot

To write a feed-animal function, we need to make a conditional branch depending on whether an animal is an armadillo or a parrot. This works with instance?. (instance? C x) tests whether x is an instance of class C, particularly whether x belongs to record type C:

(defn feed-animal
  "Feed animal."
  [a]
  (cond
   (instance? Dillo a) (feed-dillo a)
   (instance? Parrot a) (feed-parrot a)))

With that, we have all the components we need for working with new types for compound data: constructor, selectors, and predicate. Enough for today!

Functional Lenses

David Frese — Wed, 15 Oct 2014 00:00:00 UTC

Lenses are a functional abstraction that have proven very useful to us in several projects. With them, you can very effectively define, query, and especially modify complex properties of larger data structures. Lenses make properties first-class citizens that you can abstract over and combine.

This article aims to show what lenses are and how to use them. The programming language used is Clojure, which we currently use in a lot and enjoy. Some tutorials for the language can be found, for example, here.

Motivation

As a motivating example, let‘s imagine we have a simple phone book as a data structure of the following type:

(def book-1
  {"Mike" #{[:work "071170709468"] [:home "07071xxx"]}
   "David" #{[:home "07121xxx"]}})

Thus, the phone book is a map with names as keys and a set of entries as values. Each entry consists of a tuple of the entry type and a string with the phone number itself.

As an example task, let‘s imagine two functions: one checks whether a specific entry exists for a name, the other adds an entry. It shouldn‘t matter whether a name is already in the phone book or not:

(defn has-entry? [book name kind number] ...)
(defn add-entry [book name kind number] ...)

What are Lenses?

First of all, the analogy with real lenses is this: you hold a lens in front of something large and see a smaller part of it. From a programming perspective, it‘s primarily about extracting a value from a data structure using a lens. In Clojure, you could define it like this:

(defprotocol Lens
  (yank [this data]))

This defines lenses as a protocol that can be implemented for different types by defining a function yank with an additional parameter data for values of this type. The first parameter of a protocol function in Clojure is always the concrete value of the respective type, and this is an appropriate name for it.

But that alone wouldn‘t be worth mentioning, of course. The crucial thing is that a lens also allows modifying the value it focuses on! Modifying in functional programming means creating a new data structure that contains a new value at the focused location. So we add another function shove for inserting a new value:

(defprotocol Lens
  (yank [this data])
  (shove [this data v]))

One way to create concrete lenses is to explicitly define the two functions yank and shove:

(defrecord ExplicitLens
  [yanker shover]
  Lens
  (yank [this data] (yanker data))
  (shove [this data v] (shover data v)))

(defn lens
  [yanker shover]
  (ExplicitLens. yanker shover))

The record type ExplicitLens has the two fields yanker and shover, and implements the Lens protocol directly with these two functions. The function lens constructs a value of type ExplicitLens.

Application

Defining the right properties as lenses is not always so easy. The first one we‘ll need for our example is the value stored in a map for a specific key. For this, we write a function member that takes a key and a default value and creates a lens that, when held over a concrete map, focuses the associated value:

(defn member
  [key & [default]]
  (lens #(get % key default)
        #(if (= %2 default)
           (dissoc %1 key)
           (assoc %1 key %2))))

Member thus takes a parameter key and an optional parameter default, and calls lens with corresponding yanker and shover functions (# together with % or %1 and %2 is Clojure‘s shorthand notation for „lambda expressions“ with one or multiple parameters.)

The yank function of the member lens returns the value matching the key (or the default value if the key is not in the map); the shove function changes the value for a key or removes the key and value from the map when we pass the default value.

With this, we can define the first interesting property of a phone book as a lens, namely the set of entries for a name, with an empty set as the default value:

(defn book-entries
  [name]
  (member name #{}))

As mentioned, we can use a lens to read and set this property. An example:

(def my-entries (book-entries "David"))

(yank my-entries book-1)
;; => #{[:home "07121xxx"]}

(shove my-entries book-1 #{})
;; => {"Mike" #{[:work "071170709468"] [:home "07071xxx"]}}

This shows that such properties are now first-class: (book-entries "David") creates a lens for my entries in a phone book! You can bind this value to a name as here, or pass it to other functions and process it further - we‘ll come to that below.

By the way: Because member completely removes a map entry that corresponds to the default value, the new phone book created by the last expression no longer contains a "David" key.

We now also need to step into the set of entries. For this, lenses of the following type are helpful:

(defn contains
  [v]
  (lens #(contains? % v)
        #(if %2
           (conj %1 v)
           (disj %1 v))))

The function contains takes a value and returns a lens that focuses on the boolean property of whether this value is contained in a set or not. The yank function of this lens checks whether this value is contained in a set; the shove function adds or removes a value, depending on the second argument.

Hence, for our phone book entries, we could define initially:

(defn entries-contains [kind number]
  (contains [kind number]))

And we can use it directly on the sets like this:

(def contains-my-work-number
  (entries-contains :work "071170709475"))

(yank contains-my-work-number #{[:home "07121xxx"]})
;; => false

(shove contains-my-work-number #{[:home "07121xxx"]} true)
;; => #{[:home "07121xxx"] [:work "071170709475"]}

Now we want to combine the lenses for map entries and those for sets. In this case, we want to chain them together, or layer them, to stay with the metaphor. The combined lens should first apply the yank function of the lens for a map entry when reading, and then apply the yank function of a lens for the set entry on the resulting set. When writing, the shove function should correspondingly work the other way around. If we apply some wishful thinking, we need a combinator called >> that can do the following:

(def contains-my-work-number-for-me
  (>> my-entries contains-my-work-number))

(yank contains-my-work-number-for-me book-1)
;; => false

(shove contains-my-work-number-for-me book-1 true)
;; => {"Mike" #{[:work "071170709468"] [:home "07071xxx"]}
;;     "David" #{[:home "07121xxx"] [:work "071170709475"]}

And indeed, it‘s not so difficult to define this extremely useful combinator:

(defn >>
  [l1 l2]
  (lens (fn [data] (yank l2 (yank l1 data)))
        (fn [data v] (shove l1 data (shove l2 (yank l1 data) v)))))

The yanker first reads the value that lens l1 focuses on in data, and then reads from it what lens l2 has defined. The shover also first reads what l1 points to in the existing data, lets l2 make its changes in it, and finally sets the result back to the appropriate „location“ as defined by l1.

(The extension to more than two lenses is not difficult either.)

Finally, let‘s come to the two functions on phone books that we set ourselves as a task at the beginning:

(defn has-entry? [book name kind number] ...)
(defn add-entry [book name kind number] ...)

If we define a lens for the presence of an entry:

(defn book-contains [name kind number]
  (>> (book-entries name)
      (entries-contains kind number)))

Then the two functions are simply the application of this lens:

(defn has-entry? [book name kind number]
  (yank (book-contains name kind number) book))

(defn add-entry [book name kind number]
  (shove (book-contains name kind number) book true))

How you could now define a remove-entry is certainly obvious.

Laws

Not everything that conforms to the Lens protocol above should be considered a lens. The following rules, or laws, make lenses truly meaningful:

You always pull out what you put in:

(yank l (shove l d v)) == v
Putting in what you pulled out doesn‘t change anything:

(shove l d (yank l d)) == d
Putting in twice is the same as putting in once:

(shove l (shove l d v) v) == (shove l d v)

The lenses from this post satisfy all laws.

Summary

With lenses, modifiable properties of data structures can be defined very precisely and with minimal redundancy.

It‘s very practical, for example, together with our web client library reacl, as already mentioned in an earlier article. But we have also successfully used lenses in Xtend on a mutable Java data structure.

Addendum: Lenses for Clojure and ClojureScript are, in slightly different form, part of our active-clojure library.

Build Systems with Haskell

Stefan Wehr — Thu, 16 Jan 2014 00:00:00 UTC

Welcome to the New Year! We‘ll get started right with the first article of 2014 on a build system written in Haskell called shake.

Larger software projects (almost) all use a build system to automatically create a finished software product from source code. This includes, for example, compiling source files, linking object files, generating documentation, or assembling distribution archives.

This blog article provides an introduction to the shake build system, written in Haskell. This system has the advantage that dependencies between build artifacts can arise dynamically, i.e., while the build system is running. With make, probably the best-known build system, dependencies must be known before invoking the build system, which in practice often leads to limitations and problems.

We use shake at our company to compile our product Checkpad MED. Here shake plays to its full strengths, as an important component of the Checkpad infrastructure is code generation. Thanks to dynamic dependencies, it is possible to compile the program that generates the code, generate the code itself, and compile and link the generated code with a single invocation of the build system.

Neil Mitchell, the author of shake, developed a variant of the tool for use at Standard Chartered to efficiently compile really large software projects. Details on this as well as detailed information on shake‘s internal architecture can be found in this article.

As an example, we develop a build system as an example for a project written in the C language. To make the example more interesting, one of the involved .h files is generated, with the source code of the generator itself being part of the project. This roughly corresponds to the setup we have in the Checkpad project. For simplification, we‘re using a project written in C here, since the dependencies when compiling Haskell source code are significantly more complicated than for C source code. shake itself has no assumptions about the programming language(s) of the project being built. For example, it is also possible to compile Java projects with shake. However, such an endeavor would be made more difficult by the fact that when compiling Java files, it is difficult to predict which .class files will be generated from a .java file.

Let‘s start with a very simple helper function for extracting #include files from C source files. The function reads a .c file and returns the filenames of all lines beginning with #include ".

cIncludes :: FilePath -> Action [FilePath]
cIncludes x =
    do s <- readFile' x
       return $ mapMaybe parseInclude (lines s)
    where
      parseInclude line =
          do rest <- List.stripPrefix "#include \"" line
             return $ takeWhile (/= '"') rest

The type Action is a monad defined in Shake in which all build actions are executed. An essential aspect of the Action monad is the tracking of dependencies. For example, the readFile' function used above is a wrapper around the readFile function from the standard library, which in addition to reading the file also remembers a dependency on the file.

Now we come to the actual rules of our build system. Rules are defined in the Rules monad. Usually the Shake *> operator is used for this, which has the following type signature:

(*>) :: FilePattern -> (FilePath -> Action ()) -> Rules ()

We‘ll see this operator in action right away:

rules :: Rules ()
rules =
    do "*.o" *> \out ->
           do let c = replaceExtension out "c"
              need (cIncludes c)
              system' "gcc" ["-o", out, "-c", c]

The above rule describes how to create a .o file. The first argument of *> is the pattern that must match the output file, the second argument is a function that creates this output file. In the above rule we see that the .c file belonging to the .o is compiled by gcc. Beforehand, dependencies on the header files referenced in the .c file are dynamically introduced using need. If you are familiar with make, you have surely noticed that something like this doesn‘t work directly in make but only with tricks, and these tricks often have their price.

Usually, dependencies on header files are satisfied trivially, since header files are typically not generated by the build system. For our example, however, we assume that there is a header file Auto.h that is generated by a program Codegen:

       "Auto.h" *> \out ->
           do need ["Codegen"]
              system' "./Codegen" [out]

Finally, there are two more rules for the two binaries Codegen and Main:

       "Main" *> \out -> buildBinary out ["Hello.c", "Main.c"]
       "Codegen" *> \out -> buildBinary out ["Codegen.c"]

Below is the buildBinary function that creates a binary from given .c files. First, a dependency on the corresponding .o files is specified using need, which causes the .c files to be compiled into .o files using our very first rule. Then the .o files are linked by invoking gcc.

buildBinary :: FilePath -> [FilePath] -> Action ()
buildBinary out cs =
    do let os = map (\c -> replaceExtension c "o") cs
       need os
       system' "gcc" (["-o", out] ++ os)

Note that, unlike, say, make, shake doesn‘t use its own language to express dependencies and build rules. Instead, Haskell is simply used for both purposes! Two properties of Haskell are particularly helpful here: monads and user-defined operators like *>. In other words: to specify build rules and dependencies, we use in shake a small special-purpose language embedded in Haskell, i.e., a DSL. The special-purpose language is only special insofar as it uses certain operators and monads; it is still Haskell. Therefore, it is also easily possible to implement, for example, the extraction of header files included via #include using „normal“ Haskell.

Finally, here‘s the main function, which essentially specifies the targets to be built by calling the want function and passes the just-specified set of rules to shake.

main :: IO ()
main =
    do args <- getArgs
       shake shakeOptions $
          do let targets = if null args then ["Main"] else targets
             rules
             want targets

If you want to try out our little build system, you can find an example project here. Have fun experimenting!

For all Haskell programmers among you who have built your projects with cabal until now: stick with cabal if cabal‘s functionality has been sufficient for you so far. The use cases of shake and cabal are actually quite different: cabal can build Haskell projects and can invoke all the necessary tools for that. As a developer, you have little work with it, although some things like code generation are difficult to realize with cabal. shake, on the other hand, is a programming language-independent framework for build systems, so you have to define all the rules completely yourself, which is not necessary at all with cabal. In return, with shake you have significantly more flexibility than with cabal.

So, that‘s it for today. We have seen, at least to some extent, that shake allows the creation of very powerful build systems. As long as your projects don‘t require complicated build rules, using shake is certainly overkill. But as soon as things get a bit more complicated, its use can be worthwhile. In the paper mentioned above, you can find many more details about shake, in particular a comparison with other build systems.

funktionale-programmierung.de // Funktionale Programmierung

Functional Programming by Example: Refactoring a TypeScript/SolidJS Application | Part 1

Why an Audio Player?

Why SolidJS?

Approach

Audio Player: The Hidden-Model Version

Making Illegal States Unrepresentable: Modeling Intent

Writing the Model Down

Making Illegal States Unrepresentable: Implementation

Writing the Model Down (in Code)

Explicit Model Transitions

TypeScript Type Narrowing in SolidJS

The program für BOB 2026 on March 13 is up!

Talks

Tutorials

Registration

On the Testability of Pure Functions

There are impure functions that are perfectly testable

There are pure functions that are poorly testable

The substance that guarantees testability

The Call for BOB 2026 is out!

BOB 2026

Spring Boot with Scala

Dependency Injection

Annotations

Spring Boot

Example Application

Conclusion

Data Modeling with Sums and Products

Scenario

Products and Sums

Code for Sum and Products

Sums and Products Gone Wrong

Sums and Products and the Open/Closed Principle

Sums and Products in Various Languages

Kotlin

C#

Racket/Teaching Languages

Clojure

Scala

F#

Swift

Rust

Typescript

Terminology

Discussion

A Library for Configurations

Configuration

Configuration Schema

Validation and Normalization

Accessing Configuration Settings

Lenses on Configuration Settings

Projection of Configuration Settings

Profiles

Conclusion

Monads in Kotlin

The Option Type

Coroutines and Continuations

Monad Syntax as a Kotlin DSL

Data Conversion with Lenses

Data Structures

Records and Lenses

Bookmarks

Data Conversion

Projection Lenses

Projection Lenses for Records

Mixed Data

Conclusion

Looking Beyond

Functional Validation in Kotlin

Premise

Validation in Functional Programming

Pragmatics

Composition

Conclusion

Event Processing with Riemann

Riemann

Events

Streams

Tests

The `Option` Type

Everything is a `fold`