Data Conversion with Lenses

Lenses are an important concept in functional programming with great practical utility. Several years ago, we already introduced functional lenses here on the blog. Since then, we have continuously expanded our use of lenses in our daily work. Today we want to show how we use lenses as bidirectional transformations and how they help us convert between data representations.

We implement the example for this article with our comprehensive and freely available Clojure library called Active Clojure, which we use in all our Clojure projects. This library includes implementations of lenses and Records, the latter of which is for representing compound data in Clojure. The combination of lenses and records is particularly helpful.

Data Structures

Every program processes data. Structuring this data and finding suitable data models is one of the most important tasks of good and successful software development. We want to turn data structures into types in our programs; for compound data, records are suitable, see also an older blog post. As an example, let‘s consider a data definition for a bookmark manager of a web browser that links descriptions with URLs. A URL consists of a protocol, a hostname, an optional port number, and a path:

(define-record-type URL
  make-url
  url?
  [protocol url-protocol
   host url-host
   port url-port
   path url-path])

The above record definition provides a constructor called make-url that expects as arguments values for the four fields protocol, host, port, and path. Additionally, the record definition provides a predicate url? that checks whether an argument is of type URL, and for the four fields the selectors url-protocol, url-host, url-port, and url-path. Thus the expression

(def url-1 (make-url "https" "funktionale-programmierung.de" 443 "/"))

binds an instance of a URL for the internet address of our blog to the name url-1. Access to the host field is possible with the selector

(url-host url-1)

and returns

"funktionale-programmierung.de"

Records and Lenses

Actually, the selectors are not just simple functions that return the values of the fields; they are lenses that focus on these values. From lenses you can read out the focused value with yank, so you can also write this for the above access:

(lens/yank url-1 url-host)

Since every lens performs an implicit lens/yank when applied to an argument (namely the data structure), we will also subsequently omit lens/yank for all other lenses and use the more compact, clearer selector notation.

With lenses you can also modify the focused values in data structures with shove:

(lens/shove url-1 url-host "active-group.de")

The above expression returns a new URL record that points to the homepage of Active Group (shove does not mutate the old URL record and therefore has no side effect).

Analogous to yank, there is also a more compact notation for shove. For this, you apply a lens to two arguments: the data structure and the new value:

(url-host url-1 "active-group.de")

is synonymous with the above shove expression¹.

Bookmarks

Now we continue with the data definition of our bookmark manager. A bookmark is a link that consists of a description and a URL:

(define-record-type Link
  make-link
  link?
  [description link-description
   url link-url])

Moreover, our bookmark manager consists of a list of all our bookmarks:

(define-record-type Bookmarks
  make-bookmarks
  bookmarks?
  [bookmarks bookmarks-bookmarks])

With this we can define the following bookmarks:

(make-bookmarks
 [(make-link "Largest German-language FP blog"
             (make-url "https" "funktionale-programmierung.de" 443 "/"))
  (make-link "Functional software development"
             (make-url "https" "active-group.de" 443 "/"))])

Data Conversion

Data conversion is the transformation of structured data into another data format – usually into a simpler serializable format that is better suited for transmission or storage.

For our bookmark manager, Extensible Data Notation (EDN for short) is suitable as a data format for storing or transmitting information. This could look like this:

{:bookmarks [{:description "Largest German-language FP blog"
              :url         {:protocol "https"
                            :host     "funktionale-programmierung.de"
                            :port     443
                            :path     "/"}}
             {:description "Functional software development"
              :url         {:protocol "https"
                            :host     "active-group.de"
                            :port     443
                            :path     "/"}}]}

Data conversion is needed very frequently when writing programs. With today‘s widespread client-server architecture, constant communication and data transmission between client and server is necessary, so as a software developer, one is constantly dealing with converting data representations for daily work. This is often tedious, means a lot of typing work, and is error-prone.

Projection Lenses

The implementation of data conversions becomes much easier with projection lenses. Projection lenses are lenses that connect two lenses with each other and can thus link values in different data formats. A simple example helps understand this idea:

Let‘s first take two simple EDN data structures:

• a map

{:a 23
 :b 42}

• and a vector

[23 42]

The keywords :a and :b are simultaneously lenses that focus on the respective values in the example map. To obtain lenses on the elements of the vector, we can use the index in the vector, so here (lens/at-index 0) for the first and (lens/at-index 1) for the second element.

A projection lens that projects the value of the map for the key :a to the first position of the vector and the value for :b to the second position of the vector connects the corresponding lenses and looks like this:

(def map->vector
  (lens/projection [] {(lens/at-index 0) :a
                       (lens/at-index 1) :b}))

We pass the mapping of which lenses map to each other as a map. Note that projection lenses have a direction and need a neutral element for the target data structure. Here the direction of yank is from the map to the vector, hence the empty vector is the neutral element. By reversing the mappings and adjusting the neutral element, we can change the direction in the lens/projection call. Alternatively, there is a lens invert that reverses a lens. A reversed lens also needs a neutral element for the target data format:

(def vector->map (lens/invert map->vector {}))

Converting the map to the vector then corresponds to a yank of this lens on the map:

(map->vector {:a 23 :b 42})

returns [23 42].

And shove on an empty map and the vector or a yank on the inverted lens converts back to the map representation {:a 23 :b 42}:

(map->vector {} [23 42])
(vector->map [23 42])

Projection Lenses for Records

Records can automatically generate appropriate projection lenses to make usage even easier. This works with the optional argument :projection-lens in a record definition, which specifies the name of the binding for the generated projection lens:

(define-record-type URL
  {:projection-lens into-url-projection-lens}
  make-url
  url?
  [protocol url-protocol
   host url-host
   port url-port
   path url-path])

The projection lens into-url-projection-lens already knows the lenses of the associated record fields. We can connect these fields with the lenses of our EDN data structure by passing them in the correct order – matching the order of the fields and the order in the constructor call – to into-url-projection-lens:

(def edn->url (into-url-projection-lens :protocol :host :port :path))

We can now use this lens for conversion between record representation and EDN representation; the record projection lenses are defined with the record as target. Thus, in our case they start from the EDN data structure, so a yank on the lens

(edn->url {:protocol "https"
           :host     "funktionale-programmierung.de"
           :port     443
           :path     "/"})

returns the record data representation:

(make-url "https" "funktionale-programmierung.de" 443 "/")

and shove into the empty map converts the record back to EDN format:

(edn->url {} (make-url "https" "funktionale-programmierung.de" 443 "/"))

All of this naturally also works for nested data structures. Here are the definitions still missing for the rest of the bookmark example, first for a bookmark:

(define-record-type Link
  {:projection-lens into-link-projection-lens}
  make-link
  link?
  [description link-description
   url link-url])

(def edn->link
  (into-link-projection-lens :description
                             (lens/>> :url edn->url)))

We achieve the correct conversion of nested data structures through composing lenses using the lens/>> operator. We link :url with the projection lens for URLs defined above, edn->url.

Now we still need the list of bookmarks:

(define-record-type Bookmarks
  {:projection-lens into-bookmarks-projection-lens}
  make-bookmarks
  bookmarks?
  [bookmarks bookmarks-bookmarks])

(def edn->bookmarks
  (into-bookmarks-projection-lens (lens/>> :bookmarks (lens/mapl edn->link))))

Here we achieve the correct conversion of the nested data structures with another combination of lenses: First we compose with lens/>> and then we use lens/mapl, a lens combinator that applies the passed lens to all elements of a list.²

With this we have all the components needed and can use the elegantly defined lens edn->bookmarks to convert our complete data definition between records and EDN.

Mixed Data

Our solution is extensible: As a new feature of our bookmark manager, we want to introduce folders for better organization. We want to allow folders everywhere we previously allowed individual bookmarks. We now have mixed data: elements of a bookmark list can be both links and folders.

A folder consists of a name and a list of bookmarks:

(define-record-type Folder
  {:projection-lens into-folder-projection-lens}
  make-folder
  folder?
  [name folder-name
   bookmarks folder-bookmarks])

Even within a folder, all bookmarks can be either links or folders again, so we are dealing with a mixed and mutually recursive data definition here. For dealing with mixed data, there is a built-in lens combinator union-vector. And for the mutual recursion, we don‘t need to handle it differently in the conversion to EDN than we have to do in Clojure anyway. In our bookmark->edn lens, we reference a value edn->folder that is still to be defined, so we must declare the name edn->folder beforehand and delay the evaluation of the value with the lens defer, passing edn->folder as a Clojure variable object:

(declare edn->folder)

(def bookmark->edn
  (lens/union-vector [link? (lens/invert edn->link)]
                     [folder? (lens/invert (lens/defer #'edn->folder))]))

As arguments, lens/union-vector receives a list of alternatives, where an alternative is a pair of a predicate and a lens that matches the data format to which the predicate applies. The specified lens thus uses a projection lens link->edn for links when the predicate link? matches; and a projection lens folder->edn when it matches the predicate folder?. Since we are projecting starting from the records, we must invert the projection lenses:

(def link->edn
  (lens/invert edn->link))

(def folder->edn
  (lens/invert edn->folder))

The lens edn->link remains unchanged from above, but the lens edn->folder is still missing. For this we need another conversion, since, in a folder, bookmarks can again consist of links and folders. Therefore we first define a projection:

(def edn->bookmarks-with-folders
  (into-bookmarks-projection-lens (lens/>> :bookmarks
                                           (lens/mapl (lens/invert bookmark->edn)))))

And with this we have all the parts needed to use the projection lens of the Folder record for the definition of edn->folder:

(def edn->folder
  (into-folder-projection-lens :name
                               (lens/>> :bookmarks
                                        (lens/mapl edn->bookmarks-with-folders))))

We can now try this out: converting to EDN and back to the record data representation returns the original value:

(edn->bookmarks-with-folders
 (bookmarks-with-folders->edn
  (make-bookmarks [(make-folder
                    "Blogs"
                    (make-bookmarks
                     [(make-link "Largest German-language FP blog"
                                 (make-url "https" "funktionale-programmierung.de"
                                           443 "/"))]))
                   (make-link "Functional software development"
                              (make-url "https" "active-group.de" 443 "/"))])))

We implemented the extension without having to change previously written code. Great!

Conclusion

Projection lenses are a suitable abstraction for conversions between different data formats. Their use is simple, not very error-prone, extensible and easily maintainable, thus facilitating software development.

Looking Beyond

This way of combining lenses to transform data is closely related to Pickler Combinators by Andrew J. Kennedy.

The presented record projection lenses are also excellent for interaction with the lenses in our configuration library active.clojure.config. We will present this in another blog post.