In my previous blog post series Approaching The Web After Tomorrow I've talked about how I tried to implement the idea that Nikita described in his blog post called The Web After Tomorrow.
In part 1 of Approaching The Web After Tomorrow I talked about how we used DataScript, Datomic and Clojure(Script) to build a real-time app for our SaaS product Storrito.com and what performance challenge the first implementation yielded. Part 2 was about some ideas I had to solve this performance challenge. Finally Part 3 was about what difficulties you can avoid by creating your API for exclusively one frontend app and that big organizations tends to separate developers into frontend and backend developer teams.
For this blog post series I've prepared an example to introduce you to an approach I named "db-view". During my search for a good example use case, I noticed again the deep division that splits our industry into frontend and backend software development. You probably all know the de-facto standard example of almost every frontend framework: the TodoMVC, an app to manage todo entries. New to me was that there is also a collection of example backend implementations for this todo app: todobackend.com. Interestingly enough it is hard to find examples that implements both, the frontend and the backend. This is also the case for another popular collection of examples: realworld.io or how they call it "the mother of all demo apps", an exemplary fullstack clone of Medium.com. This project even enforces the segregation of the frontend and the backend by requiring that all implementations follow the same API specification.
This leads to the benefit that you can freely exchange the frontend and the backend implementations. But this segregation might not yield the optimal trade-offs for your small developer team, that tries to implement your current project in time and budget. Such a generic API has dozens of requirements, which are hard to solve (see Part 3 of the previous blog series). Furthermore your API endpoint becomes more complex the more apps and systems use it, since all of them might need a slightly different set of database fields or functionality from your server.
Developers know the benefits of everything and the trade-offs of nothing.
Rich Hickey (source)
The db-view approach choose different trade-offs in comparison to a REST API. In essence you decide to develop your backend API endpoint with a single frontend app in mind. All returned data is tailor-made for your particular frontend implementation. Instead of implementing a generic todo entries API endpoint, you return the data in a shape that is designed specifically for your todo list UI component. Thereby this shape can stay simple, since it doesn't have to satisfy and intertwine the requirements of multiple UI parts (or different apps).
You only need a single API endpoint for the read-related (meaning no side-effects) part of app. Let's use the URL path /db-view/get for this API endpoint. It receives a map via the request body and returns a map in the response body. EDN is used as data format for the example below:
Example request body:
{:todo/list {}}
Example response body:
{:todo/list {:todo/list-items
[{:db/id 17592186045418
:todo/title "Create an example"
:todo/done true}
{:db/id 17592186045419
:todo/title "Write a blog post"
:todo/done false}
{:db/id 17592186045420
:todo/title "Publish the blog post"
:todo/done false}]
:todo/active-count 2}
}
The corresponding UI is the one of the TodoMVC examples and it looks like this:
The request claims that server should return the :todo/list part of the db-view. The response contains the data that is needed to render the todo list UI of your frontend. The :todo/active-count is used to display the number of active todo entries, which are still not done.
To only show the todo entries that are already done, you send the request:
{:todo/list {:todo/filter :completed}}
And receive the response:
{:todo/list {:todo/list-items
[{:db/id 17592186045418
:todo/title "Create an example"
:todo/done true}]
:todo/active-count 2}
}
Once again the response data is tailor-made for the todo list UI that only shows the completed todo entries now:
Note that the :todo/active-count ("2 items left") is still shown in the bottom bar, that's why it is still included in the response. Most likely your app shows multiple UI parts at once. Let's say your app always displays the email address of the current user at the top of the page:
Example request:
{:todo/list {:todo/filter :completed}
:user/info {}}
Example response:
{:todo/list {:todo/list-items
[{:db/id 17592186045418
:todo/title "Create an example"
:todo/done true}]
:todo/active-count 2}
:user/info {:user/email "max@example.com"}
}
Here the :user/info entry in the request tells the server to include the information of the current user in the response body. The :user/info entry in the response contains the :user/email to display it at the top of the page.
Let's pretend multiple people are using this todo app simultaneously. When a user adds a new todo entry, how do the other users receive the update. In the case of a "classic" web application the other users would need to click the browser's refresh button to receive the latest state from the server that includes the new todo entry.
The db-view approach uses a similar mechanism, with the difference that the user do not have to press the refresh button, a kind of auto-refresh if you like. While a "classic" web application would transfer the arguments to the server via the query parameters of the URL, a db-view frontend stores these parameters in a map (JSON or EDN) and sends them via a POST request to the server. To refresh our app with the latest state of the server, we just repeat this POST request with the same parameters as last time.
To receive the update with the new todo entry, the dumbest implementation would just poll the server every second. Of course normally you would use long-polling, websockets, server-sent events or a service like Pusher to send a signal to the client. This signal does not contain any payload, it just tells the client to do a refresh.
To avoid a full page refesh, the frontend needs to use a virtual dom library (like React). The example for this blog post series uses Reagent (also React-based) to implement the UI. The complete frontend app state is held in a single (Reagent) atom. The input parameters for the db-view are stored under the key :db-view/input in the app state and are send as request body to the /db-view/get API endpoint. The response body (also a map) is stored under the key :db-view/output in the app state. From there the UI components can access the data that has been retrieved from the server. As soon as the app state is updated Reagent will rerender the corresponding UI components and the user will see the most recent state.
In the elixir community a related approach called Phoenix LiveView is emerging that does even more on the server-side, since it also does a kind of virtual dom diffing on the server-side. The trade-off is that you can implement most of the app on the server-side, but you need to store the client state on the server and deal with the latency. Luckily Clojure has a good story for client-side development due to ClojureScript. Therefore you can at least use the same programming language for the frontend and the backend. By doing more on the server-side Phoenix LiveView and db-view both reduce the number of pitfalls due to the fact that you can not trust the client (e.g. an attacker can modify the client source code or send fake requests to your API).
More on that in the next blog posts, which will also provide a detailed look on the source code of the db-view example todo app. Furthermore we we will also learn how to implement side-effects in the context of a db-view application.