Neil: Bohemian Rhapsody for Five Channels

This post is dedicated to all the brave souls who chose Janet as a programming language for Advent of Code 2021.

Raison d'etre

When I emerged from the dire straits of the first Lockdown (as described in How I become Janet) with the knowledge that Janet was helping me survive, I started to search for the problem I could solve. I had already developed some ba­sic tools under the Good Place umbrella, but I am mostly the programmer for business needs. It did not take long before I started with a time tracker. As a freelancer, I used many of them, but I was never happy with any of them. So two days before independence day of the year of the corona strike, I have started with a simple RPC server and CLI client for usage from the terminal. In a week, I had encrypted communication with the bare-bones server, and I started to track my work with it. By the end of the month, I had a TUI dashboard. After six hundred sixty-six commits, two significant rewrites, and more than five hundred hours of work poured into the development, I feel we're, together with Neil, ready to present ourselves to the cruel world. And with the code samples, no less.

As I have already men­tioned, I did two big reimaginations of the software stack, which I am using for (not only) Neil's development. Today I will skip the storage, with all the dramatic scenes of rejection of the status quo and corporate code culture, cause I would like to talk about the channel story.

Channels

The event loop as part of the virtual machine was the most significant change to Janet since I started to hack on it. I have already done pre­liminary description in the ode series, so just to recap:

• The event loop is written in C as part of the virtual machine.
• Core concept of the language present from almost the start, a fiber, is the basic building block of the event loop.
• Thread functionality was moved under an event loop, with copy only data transfer.
• You can use channels for the communication to and from the event loop.
• You can give to and take from channels asynchronously.
• Channel could be set as a supervisor for fiber or thread you put on the event loop. And from that fiber, you can simply give-supervisor any value.

Stereo

My take on reactive streams is called Shawn, as I had met them when I worked with the potok library in ClojureScript. But instead of mimicking all RX features, I stripped the idea down to the naked tool assembled from fundamental Janet constructs. Same as with the potok, you have a store (Shawn) with the state (envelope), and you can transact (confirm) events (acts).

Shawn instance is a table with a prototype containing everything needed for its functionality. An envelope is just a vanilla Janet table, usually a tree-like structure, you supply when initializing Shawn. Acts are also Janet tables with required methods, called by Shawn when it processes them.

(def Shawn
  ```
  Shawn prototype. It has two public methods:

  * (:confirm shawn & acts): confirms given Acts.
  * (:admit shawn): waits for Shawn to unzip all cocoons.
    Returns the array of the envelope and all cocoons results.
  ```
  @{:confirm confirm
    :admit admit
    :_tide @[]
    :_flow (ev/chan 10)
    :_thread-flow (ev/thread-chan 10)
    :_cocoons 0
    :_snoops @[]
    :_fall-tide _fall-tide})

Figure 1: Shawn prototype

Acts have four possible flavors, and you can use one or all for a particular need. The flavors are:

• Spy Act has to have :spy method, called with the envelope as an argument, and must return the Snoop or array of Snoops.
• Update Act has to have :update method, called with the envelope, and can change the data in the envelope. NB: you can change an envelope at any type of Act, but I will look upon you if you do!
• Snoop has to have :snoop method, which shawn calls after every :update method call of every UpdateAct.
• Watch Act has to have :watch method that, according to state, generates new events.
• Effect Act has to have :effect method that does all the heavy lifting of communication outside of the Shawn.

(defn start-task [id]
  (make-act
    {:update
     (fn [_ e]
       (def {:brush brush} e)
       (unless (running-task brush)
         (def task (load brush id))
         (stamp-ident brush (task :project))
         (-> task
             (update :work-intervals
                     array/concat @{:start (os/time)})
             (put :timestamp (os/time)))
         (:paint brush id :task/running)))
     :watch (fn [&] Dry)}
    "start-task"))

Figure 2: Example of Act with both :update and :watch

In this example, :update method does all the business required operations and then saves the result into envelope e. brush a database instance reference in an envelope, so every change to brush is promoted into the envelope too. :watch the method emits Act for saving the current state of brush to the disk.

The most intriguing yet powerful is the watching and snooping. With this mechanism, you can easily serialize events in your system with all the powerful tooling of the standard library. But where is anything async, you may ask? I am glad you did. Again, the answer is in watching cause you can emit a "special" fiber called Cocoon in Shawn's parlance. These Cocoons will receive one of the two types of supervisor channels:

• Flow is for supervising fiber-based Cocoons.
• Threaded Flow is for supervising thread-based Cocoons.

The optional last part is calling :admit method on Shawn to wait for it to finish. After all the Cocoons finish, this method returns all Cocoons' products and final envelope products.

I need to say that it can be infinite, and Shawn will never admit it, just like in the TV series. It is because as a special kind of product Cocoon can emerge WatchAct which is confirmed immediately, and the machinery goes from the beginning. You have two channels to moderate the whole buzz of the fibers, Acts and Cocoons. And believe me, there can be a lot of them inside Shawn.

(defn admit
  ```
  Blocks until all Cocoons on the Shawn supervisor channel finishes.
  Returns array, where the first member is the final envelope
  followed by all results from cocoons.

  This function is called when you call :admit method on Shawn.
  ```
  [shawn]

  (def res @[])
  (defn dec-cocoons-add-res [val]
    (update shawn :_cocoons dec)
    (array/push res val))
  (while (pos? (shawn :_cocoons))
    (match (last (ev/select (shawn :_thread-flow) (shawn :_flow)))
      [:ok (cocoon (fiber? cocoon))]
      (dec-cocoons-add-res (fiber/last-value cocoon))
      [:ok val]
      (dec-cocoons-add-res val)
      [:yield cocoon]
      (array/push res (fiber/last-value cocoon))
      [:emergence acts]
      (:confirm shawn ;(seq [a :in acts] (make-act a)))))
  (array/insert res 0 (shawn :envelope))
  res)

Figure 2: Shawn's admit method

The algorithm of admitting is quite simple, just count the remaining cocoons and take values from the supervisor channels. In this case with ev/select, as there are two channels to take from.

Net Trinity

The next three libraries are all network servers.

Soulmate

My first Library of a Good Place and my attempt at the web is Chidi. The first incarnation used a lot of generated code and entirely run-of-the-mill architecture. When ev came, I scratched it all and started anew. Chidi is now a modu­lar library for building the http servers. And not only http, as you will see later. The server module contains everything to serve the http requests with respo­nse strings. But not­hing more, even parsing the request and routing is in the middleware module. All status codes and response constructs are in the response module. If it sounds familiar welcome, if it does not, do not be afraid. Janet, sharing a big chunk of Lisp philosophy, has everything you need for such tasks. Just functions everywhere.

The part of the code interesting here is the life cycle of the connection. After accepting it from the network, Chidi creates a fiber in which the process of retrieving the request, processing it, making some business logic, and printing the response is done. It has many advantages. One of them is assigning the super­visor channel, which you have a chance to process if you want to. Or use the default provided functions. But that would be a loss of the chance, mind you.

(defn default-supervisor
  ```
  It takes `chan` as the supervising channel of the server
  and `handling` as the handling function.
  This supervisor is used by default if you do not
  provide your own to `start`
  ```
  [chan handling]
  (forever
    (match (ev/take chan)
      [:close connection] (:close connection)
      [:conn connection]
      (ev/go
        (fiber/new
          (fn handling-connection [conn]
            (setdyn :conn conn)
            (handling conn)) :tp) connection chan)
      [:error fiber]
      (let [err (fiber/last-value fiber)]
        (unless (or (= err "Connection reset by peer")
                    (= err "stream is closed"))
          (debug/stacktrace fiber err)
          (def conn ((fiber/getenv fiber) :conn))
          (protect (:write conn ise))
          (:close conn))))))

Figure 3: Chidi's default supervisor

Here is the main mechanism of the handling signals on the supervisor channel. Just forever take from the supervisor channel and match it to the right action.

Daemon

The next part of the web story is the server for handling web sockets. For that purpose, GP includes Trevor library. As mentioned above, Chidi's core design allows you to serve any network connection, and with the correct handler and supervising function, you can serve websockets. The only other task Trevor fulfills is the websockets handshakes and creation of response.

(defn supervisor
  [chan handling]
  (forever
    (match (ev/take chan)
      [:emergence acts] (cocoon/emerge ;acts)
      [:close connection] (:close connection)
      [:conn connection]
      (ev/go
        (fiber/new
          (fn [conn]
            (setdyn :conn conn)
            (handling conn))
          :tp)
        connection chan)
      [:error fiber]
      (let [err (fiber/last-value fiber)
            conn ((fiber/getenv fiber) :conn)]
        (unless (one-of err
                        "Connection reset by peer"
                        "stream is closed")
          (protect (:write conn (trevor/text (json/encode {:error err}))))
          (ev/give-supervisor :close conn))))))

Figure 4: Trevor's supervisor in Neil

As you can see the code is very similar to Chidi's default supervisor, with the difference of including :emergence branch, which just reemerges acts to Shawn's Flow.

Monster

The last dynamic part of Neil is RPC. The library for developing it in a Good Place is named Hemple. It is almost a verbatim copy of the spork/rpc server with just a couple of differences: all the communication is minimized with miniz and encrypted with jhydro. And the most significant change in implementing Chidi's supervising protocol.

Hierarchy

As we have found all the channels, the last task is to show how they are coerced into a hierarchy. The structure is straightforward. At the root is Shawn's flow cause I am not using threaded Cocoons for Neil. When you emerge Act from your business code in the server, you have a chance to reemerge it to Shawn, where all the state is stored and mutated, as can be seen in Figure 4. So by two layers of channels, you can control every aspect of the whole application functionality.

Also by running everything in fibers, you have a standard way of dealing with errors and failures, without crashing the whole thing.

Conclusion

This is the biggest dive in any part of the GP code. I can understand if it is not that clear how the whole thing works, but I hope you now have a clearer idea of how Janet's event loop and channels work.

PS: you can be startled, or even disturbed by the title. Just to clear it up: the part of Europe I am living was called Bohemia till the end of the second world war (Deutsche Böhmen), so I played that card. If you still feel uneasy about it go listen Mixmaster Moris