Chapter 3 Major internal classes
targets relies heavily on object-oriented programming, and the following subsections describe the major classes of objects relevant to orchestration and branching. Objects that appear once per workflow are instances of R6 classes. To maximize performance, target-specific data structures are lower-tech classes: simple environments with formal constructors, helpers, and validators.
3.1 Algorithm class
An algorithm in targets is an abstract class that represents how to iterate through the pipeline target by target. Different algorithms describe different kinds of deployment: for example, execution in the main process on the host machine versus parallel execution on a cluster. Every algorithm has a scheduler object and a pipeline object.
3.2 Pipeline class
A pipeline is a wrapper around a collection of targets, and it is responsible for the initial reasoning about the topology of the pipeline before runtime. Pipelines express their reasoning by producing static graphs and scheduler objects early on. In addition, pipelines contain new buds and branches created dynamically during the pipeline.
3.3 Scheduler class
Whereas pipelines are responsible for static topology, schedulers are responsible for dynamic topology. Schedulers know
- The upstream and downstream neighbors of each target.
- The progress of each target, e.g. queued, running, or finished.
- How many upstream dependencies need to be checked or built before a target is ready to run.
To meet these responsibilities, the scheduler is composed of three smaller objects:
- A graph object.
- A progress object.
- A priority queue.
3.4 Graph class
The graph class keeps track of the upstream and downstream neighbors of each target. The scheduler adds edges to the graph when new targets are created dynamically.
The graph is implemented as two adjacency lists: one for upstream edges and another for downstream edges. For the purposes of powering a pipeline, we find this low-tech structure to be more efficient than igraph in our situation where we repeatedly query the graph and the number of nodes is small. (Transient igraph objects, however, are created for validation and visualization purposes.)
3.5 Progress class
The progress class keeps track of the state of each target: queued, running, skipped, built, outdated, canceled, or errored. To accomplish this, the progress object maintains a counter object for each category.
3.6 Queue class
The queue class is a priority queue, essentially a wrapper around a named integer vector of ranks. For targets’ purposes, the rank of a target is the number of unmet dependencies so far, minus a per-target priority value in the interval [0, 1) to control the order in which targets are dequeued. As the pipeline progresses, the queue is checked and modified periodically as dependencies are met. The next target to build is the lowest rank target such that -1L < rank <= 0L. Branches are pushed to the queue when they are dynamically created.
3.7 Counter class
A counter is an efficient abstraction for keeping track of target membership in a category. A counter stores the number of targets in the category and a hash table with the names of those targets. Counters are used to efficiently keep track of runtime progress (e.g. running, queued, or built) as well membership in the queue.
3.8 Target class
A target is an abstract class for a step of a pipeline. Each target is a composite of intricate sub-classes that keep track of commands, in-memory dependencies, storage, settings, and some aspects of build behavior. As its name implies, the targets package pushes most of its conceptual complexity to the target level in order to decentralize the architecture and make it much easier to reason about the pipeline as a whole.
There are multiple sub-classes of targets, and the different behaviors of different sub-classes drive the orchestration and branching of targets. The inheritance hierarchy is as follows.
- Target
- Bud
- Builder
- Stem
- Branch
- Pattern
3.9 Stem class
A stem is the most basic form of target. It is neither a pattern nor part of a pattern. However, it can dynamically create buds to assist with branching.
3.10 Bud class
A bud is a target that simply contains part of a stem’s value in memory. The purpose of a bud is to serve as a dependency of a branch when a pattern branches over a stem.
3.11 Pattern class
A pattern is an abstract class responsible for creating new branches and dynamically updating the scheduler. Stems and patterns are the only targets the user manually defines in the pipeline.