Tapestry
Next generation concurrency primitives for Clojure built on top of Project Loom
About
Project Loom is bringing first-class fibers
to the JVM! Tapestry seeks to bring ergonomic clojure APIs for working with Loom.
What are Fibers and Why do I care?
Fibers behave similarly to OS level threads, but are much lighter weight to spawn, allowing
potentially millions of them to exist.
Clojure already has the wonderful core.async
and manifold libraries but writing maximally performant code
in either requires the abstraction (channels, or promises) to leak all over your code
(as you return channels or promise chains) to avoid blocking. Furthermore you frequently have to
think about which executor will handle the blocking code.
Loom moves handling parking to the JVM runtime level making it possible for “blocking” code to be
executed in a highly parallel fashion without the developer having to explicitly opt into the
behavior. This is similar to how Golang and Haskell achieve their highly performant parallelism.
Some great further reading on the topic:
- What color is your function - A mental exploration of
the leaking of the abstraction.
- Async Rust Book Async Intro - A
good explanation of why we want async. The rest of this book is fantastic in terms of
understanding how async execution works under the hood. In
manifold and core.async
the JVM executor is mostly analogous to the rust’s concept of the Executor. In a language like
Rust, without a runtime, being explicit and “colorizing functions” makes sense, but with a
run-time we can do better.
- Project Loom Wiki - Internal design notes of Loom.
Project State
Tapestry is still pre-1.0. As the APIs in loom have stabilized so too has tapestry.
Tapestry is being used in production for several of Teknql’s projects and has
more or less replaced both clojure.core/future and manifold.deferred/future.
It is the ambition of the project to eventually drop manifold entirely, at which
point it will likely hit 1.0.
Installation
Add to your deps.edn:
teknql/tapestry {:git/url "https://github.com/teknql/tapestry"
:git/sha "d4c09e1866ab9d4988f04fca83969043b4c857f8"}
Showcase
Here is a demo of some of the basics.
Spawning a Fiber
(require '[tapestry.core :refer [fiber fiber-loop]])
;; Spawning a Fiber behaves very similarly to `future` in standard clojure, but
;; runs in a Loom Fiber and returns a tapestry.core.Fiber which implements IDeref.
@(fiber (+ 1 2 3 4))
;; => 10
;; Or, Like `core.async`'s `go-loop'
@(fiber-loop [i 0]
(if (= i 5)
(* 2 i)
(do (Thread/sleep 100)
(recur (inc i)))))
;; => 10, after aprox 500ms of sleeping
Interrupting and introspecting a Fiber
(require '[tapestry.core :refer [fiber interrupt! alive?]]')
(let [f (fiber (Thread/sleep 10000))]
(alive? f) ;; true
(interrupt! f)
(alive? f) ;; false
@f ;; Raises java.lang.InterruptedException))
Timeouts
Tapestry supports setting timeouts on fibers which will cause them to be
interrupted (with a java.lang.InterruptedException) when the timeout is hit.
(require '[tapestry.core :refer [fiber timeout! alive?]]')
(let [f (fiber (Thread/sleep 10000))]
(timeout! f 100)
(alive? f) ;; true
(Thread/sleep 200)
(alive? f) ;; false
@f ;; Raises java.util.concurrent.TimeoutException))
You can also specify a default value
(require '[tapestry.core :refer [fiber timeout! alive?]]')
(let [f (fiber (Thread/sleep 10000))]
(timeout! f 100 :default)
@f ;; => :default))
You can use dynamic bindings to set a timeout on a bunch of fibers. Note that
each fiber will have a timeout that starts from when the fiber was spawned.
(require '[tapestry.core :refer [fiber alive? with-timeout]]')
(with-timeout 100 ;; Accepts a duration or number of millis
(let [f (fiber (Thread/sleep 10000))]
@f ;; raises java.util.concurrent.TimeoutException
))
Processing Sequences
(require '[tapestry.core :refer [parallelly asyncly pfor]]
'[clj-http.client :as clj-http])
(def urls
["https://google.com"
"https://bing.com"
"https://yahoo.com"])
;; We can also run a function over a sequence, spawning a fiber for each item.
(->> urls
(parallelly clj-http/get))
;; We can using the built in `pfor` macro to evaluate a `for` expression in parallel. Note that unlike
;; clojure.core/for, this is not lazy.
(pfor [url urls]
(clj-http/get url))
;; Similalry, if we don't care about the order of items being maintained, and instead just want
;; to return results as quickly as possible
(doseq [resp (asyncly clj-http/get urls)]
(println "Got Response!" (:status resp)))
Bounded Parallelism
;; We can control max parallelism for fibers
(require '[tapestry.core :refer [parallelly fiber]])
;; Note that you can also use `with-max-parallelism` within a fiber body
;; which will limit parallelism of all newly spawned fibers. Consider the following
;; in which we process up to 3 orders simultaneously, and each order can process up to 2
;; tasks in parallel.
(defn process-order!
[order]
(with-max-parallelism 2
(let [internal-notification-success? (fiber (send-internal-notification! order))
shipping-success? (fiber (ship-order! order))
receipt-success? (fiber (send-receipt! order))]
{:is-notified @internal-notification-success?
:is-shipped @shipping-success?
:has-receipt @receipt-success?})))
(with-max-parallelism 3
(let [order-a-summary (process-order! order-a)
order-b-summary (process-order! order-b)
order-c-summary (process-order! order-c)
order-d-summary (process-order! order-d)]
{:a @order-a-summary
:b @order-b-summary
:c @order-c-summary
:d @order-d-summary})
;; You can also bound the parallelism of sequence processing functions by specifying
;; an optional bound:
(asyncly 3 clj-http/get urls)
(parallelly 3 clj-http/get urls)
Manifold Support
(require '[manifold.stream :as s]
'[tick.alpha.api :as t]
'[tapestry.core :refer [periodically parallelly asyncly]])
;; tapestry.core/periodically behaves very similar to manifold's built in periodically,
;; but runs each task in a fiber. You can terminate it by closing the stream.
(let [count (atom 0)
generator (periodically (t/new-duration 1 :seconds) #(swap! count inc))]
(->> generator
(s/consume #(println "Count is now:" %)))
(Thread/sleep 5000)
(s/close! generator))
;; Also, `parallelly` and `asyncly` both support manifold streams, allowing you to describe parallel
;; execution pipelines
(->> (s/stream)
(parallelly 5 some-operation)
(asyncly 5 some-other-operation)
(s/consume #(println "Got Result" %)))
Working with Agents
(let [counter (agent 0)]
(tapestry.core/send counter inc)
(await counter)
@a)
;; => 1
Experimental Features
Note: The following APIs are experimental and may evolve more rapidly than
the stable tapestry.core API.
Structured Scopes
Scopes provide structured concurrency for tapestry fibers — unifying lifecycle
management, shutdown policies, timeouts, and parallelism control into a single
construct.
Any fibers spawned with tapestry.core/fiber inside a scope are automatically
registered. On scope exit, all fibers are awaited (full thread termination, not
just future completion), and shutdown policies are enforced.
(require '[tapestry.experimental.scope :refer [with-scope]]
'[tapestry.core :refer [fiber]])
;; Basic scope — awaits all fibers before returning
(with-scope {}
(let [a (fiber (fetch-user id))
b (fiber (fetch-orders id))]
{:user @a :orders @b}))
;; Shutdown on failure — if any fiber throws, siblings are interrupted
;; and the error propagates from the scope
(with-scope {:shutdown :on-failure}
(let [user (fiber (fetch-user id))
orders (fiber (fetch-orders id))]
{:user @user :orders @orders}))
;; Combined options
(with-scope {:shutdown :on-failure
:timeout 5000
:max-parallelism 4}
(let [a (fiber (do-a))
b (fiber (do-b))]
{:a @a :b @b}))
Scopes nest naturally — inner scope fibers are tracked by the inner scope only:
(with-scope {:shutdown :on-failure}
(let [a (fiber
(with-scope {:shutdown :on-failure}
(let [x (fiber (subtask-1))
y (fiber (subtask-2))]
(+ @x @y))))
b (fiber (other-work))]
{:a @a :b @b}))
alts
(require '[tapestry.experimental :refer [alts]])
;; Runs all expressions in parallel, returning the first successful
;; result. Remaining fibers are interrupted.
(alts
(do (Thread/sleep 100)
:first)
(do (Thread/sleep 10)
:second))
;; => :second after 10ms
;; With options
(alts {:timeout 3000}
(fetch-from-primary)
(fetch-from-replica))
Queues
The beginnings of potentially dropping manifold support. Lightweight wrapper around java’s
BlockingQueues with the notion of closing ala manifold and core.async.
(require '[tapestry.queue :as q]
'[tapestry.core :refer [fiber alive?]]')
;; By default a queue has no buffer
(let [q (q/queue)
take* (fiber (q/take! q))]
(alive? take*)
(q/try-take! q) ;; => nil, no value available
(q/put! q :value) ;; => true, value put successfuly
@take* ;; => :value, resolved in the fiber above
(q/close! q) ;; Closing a queue will make it so no further items are accepted,
;; but previously queued items will be delivered via `take!`
(q/put! q :value) ;; Returns false
)
Advisories
None at the moment
CLJ Kondo Config
Add the following to your .clj-kondo/config.edn
{:lint-as {tapestry.core/fiber-loop clojure.core/loop
tapestry.core/pfor clojure.core/for}}
Long Term Wish List
评论