HMFoDG is a tool that tests whether you are even allowed to run UMAP and UPGMA on one piece, or whether you must split into subsets - because neither of those tools can detect the absence of a common trunk / the absence of a shared quasinorm..
UMAP and HMFoDG are not competing for the same job. UMAP draws a low-dimensional picture of neighborhood. HMFoDG computes under which norm and at which cutoff that neighborhood may be assembled at all - and whether the shared space is not an artifact of the projection.
Kinship: both start from a graph. The difference is what they do with the graph and which metric they assume.
Task:
UMAP:
How to arrange points in 2D/3D (or in a small R^d) so that local neighborhood from high dimension stays readable?
HMFoDG:
How to locally assemble contributions of different scales/norms and return a hierarchical address plus the metric of that calculation?
Output:
UMAP:
Coordinates in low dimension + (internally) a fuzzy kNN graph
HMFoDG:
Prefixes / tree, ultrametric d ~ p^{-LCP} or e^{-λ LCP}, local w, cutoff, list of transitions;
Goal:
UMAP:
Visualization, clustering “by eye”, sometimes a feature for a model
HMFoDG:
feature for a model, audit of embeddings, training order, detection of a false shared space, mixed biotech correlations;
Determinism:
UMAP:
Stochastic minimization of cross-entropy; seed, epochs, init change the picture
HMFoDG:
Loop aggregation → Fisher/entropy → prefixes → reprojection; the hierarchy from LCP is an address, not a layout
Similarities:
Locality instead of a global distance matrix. UMAP: kNN + local density scale. HMFoDG: a star around a chosen x0, neighborhood N(v). That is the difference between a statistic and what, and where, sits in the projection.
Cutoff. UMAP cuts the world beyond n_neighbors and turns distant edges into ~0 in the fuzzy simplicial set. HMFoDG cuts floors below resolution, but computes the threshold from image sharpness (entropy / Fisher), not from one global number k.
Co-occurrence / similarity graph. In biotech UMAP often sits on PCA+kNN of cells or on distance in marker space. HMFoDG wants the same carrier, i.e. what happened to the object.
UMAP builds a local graph; HMFoDG asks whether that graph may be assembled under one norm. Specifically a quasi-norm, but I will not wander into Lorentz, Minkowski and Orlicz because that needs detailed explanations - look into the notes for how this is constructed.
So I’ll cut it short. If the spectral radius doesn’t fall under the attenuation horizon, the construction refuses to compose. You did 1000x on the miscroscope and seen what happens with colors.
I will mark here a certain difference in how questions to data are posed - I simply have to know concretely what goes where, as if I needed to hand data to a GPU pipeline so it would execute a projection for me. I cannot give “statistically somewhere”; I must return a concrete value or exit the function empty.
Differences!
- One metric vs a cascade of p
UMAP takes one metric on input (euclidean, cosine, correlation, more rarely another, but only because those metrics have to be explained) and one family of kernels. Local adaptation is mainly iteration over density and local_connectivity, not a change of the aggregation exponent; that exponent follows from the given metric, which is an assumption taken from habit.
HMFoDG treats w (that is p, the norm) as a state variable: w1 amplifies the dominant. This is the answer to the question from the checklist of the previous article: why not always L2? And in practice it is almost never exactly the idealized L2 case. - Manifold vs no global space
UMAP assumes (weakly assumes, because without a proof) that the data lie on a manifold of approximately uniform local density and that it can be embedded in low dimension (from n-dim data). Hence the artifacts: an apparent sphere, apparent bridges, “clusters” that are not in the data - classic crowding / tear. Glitches at the level of z-fighting in the projection.
HMFoDG assumes from the start that there is no global space. There is a local map around x_0 (the given projection - what do I see?). When the gluing does not hold, the prefix codes have no common position. This is a formalization of what in UMAP looks like “a pretty picture that you must not read as a world map”. - Neighborhood as a set vs path as state
In UMAP an i-j edge is a fuzzy membership weight. The path between clusters in 2D is not part of the model; the layout distorts it. Very often, being completely detached from the data, we then obtain artifacts of the drawing mechanism, not information from the data. This is a common problem in metrology - did we measure the phenomenon or the ruler?
In HMFoDG the path on the graph belongs to the state, and the same two nodes have a different measure of “how many transitions there are” and a different length. In biotech this is the difference between “these two genes are close in UMAP” and “these two events are close along this trajectory of conditions”. - Scale
UMAP: one (or possibly a few) n_neighbors for the whole set. Larger k = a more “global” view, smaller = local. That is one floor of the cascade.
HMFoDG: a scale scan with Fisher information / entropy H(N(v)) [generalized under the metric; it is not the same thing, but structurally the formula is the same and admissible, though here there would be a digression on Amari]. A cutoff that is sharp on this tower vanishes on another. Absence of a trunk at this resolution ≠ absence of a trunk at all - exactly the problem that in single-cell is solved by hand-tuning n_neighbors and min_dist. Here that problem is solved automatically in the computational loop. - Hierarchy
UMAP does not return a dendrogram. Hierarchy is drawn on afterwards (HDBSCAN on UMAP coordinates, UPGMA on distances). Those are two different calculations glued by habit.
HMFoDG: NN is max LCP, the prefix tree is the hierarchy and the search structure. The ultrametric is not post-processing. (UPGMA from biotech toolkits is closer here, not UMAP.) - What is optimized
UMAP minimizes the divergence between two fuzzy graphs (high dimension vs layout). An aesthetic-geometric goal: nearby points stay nearby on the drawing.
HMFoDG does not look for a pretty R2. It looks for the least false quasi-norm of the components at a given resolution, and a flag that the shared space is false.
That is why UMAP is excellent at showing “here are islands of cells”, and bad at answering “does marker A have a p2-norm of influence on marker B in this neighborhood”. The latter is a question for HMFoDG.
| UMAP | rough counterpart in HMFoDG |
|---|---|
| metric | choice of channel / projection; does not replace w |
| n_neighbors k | size of N(v), but without one global k |
| min_dist | pressure on cluster sharpness in the layout, not an influence cutoff |
| local σ_i | a piece of what Fisher/entropy does for w and λ |
| init / seed / epochs | in HMFoDG there is no counterpart; there is no layout to randomize |
| UMAP → HDBSCAN | prefixes + LCP immediately |
| cosine vs Euclidean in UMAP | two different inputs; HMFoDG asks whether along the way they may be assembled |
UMAP is enough when the goal is a picture (cell atlas, batch QC, “did these conditions drift apart”), one metric on input is honest (e.g. PCA-Euclidean after a reasonable gene selection) and nobody reads distances between islands as influence. That is: for drawing pictures meant to be imagined by users. This matters, because drawing a picture from HMFoDG where we have different norms may sit outside the user’s intuition from lack of practice in such projective spaces. Everyone has seen a sheet of paper, everyone has seen a 3d animation or an FPS or other 3d, or uses CAD/Max/Blender. Games with a dynamic metric and dimensionality do not exist, and they would look like a madman’s dream; games like Hyperbolica are only a mild taste of directional locality.
UMAP lies in exactly the places described by the earlier article on biotech:
mixed scales (expression, dose, time, batch, rare events) stuffed into one vector and one metric,
empty spaces (most pairs of genes/events never co-occur in the data) - kNN then invents bridges,
a question about a trajectory (“the drug works at the second full moon…”) rather than about a cloud of points,
audit: through which tokens / which associations two objects sit close (RAG, embeddings, PPI).
There UMAP will show a pretty bridge. HMFoDG is supposed to return: a short or zero common prefix, w fleeing to the extreme, Fisher with no maximum in the middle - i.e. “this is not a shared space, this is a projection”.
UMAP is a projection of neighborhood onto a drawing under one metric and one k. HMFoDG is a loop that chooses the exponent and the cutoff so that this neighborhood can be assembled at all, and categorically refuses to glue when typicality breaks. They may be used together when UMAP does a fast projection and HMFoDG does the calculation of whether the islands on the screen may be read as one geometry or whether this is a glitch.
UMAP will always arrange something. It must. A kNN graph + layout cannot return “no shared space”. When typicality breaks, the algorithm still glues islands with bridges, because n_neighbors requires neighbors and cross-entropy requires a picture. Those are exactly forced correlations, very pretty, repeatable on the same set, and absent as influence.
HMFoDG asks about a condition that UMAP does not check: whether around the chosen x_0 there is a trunk at all, onto which one quasi-norm may be laid.
In practice this means “inspect the database with HMFoDG before you draw UMAP”. On the same carrier (the same vectors / the same co-occurrences) we look not at 2D, but at signals from the loop:
| HMFoDG signal | What it does to UMAP |
|---|---|
| LCP between pairs ~0, codes with no common position | "Bridges on the drawing are addressing of emptiness, not a relation" |
| Fisher / entropy with no maximum in the middle, smear at the edge | n_neighbors and σ_i will twist the scale in noise |
| k_eff jumps when x0 or the prefix floor changes | A different center / a different subset of markers will give a different “atlas” because this is not a manifold |
| Cutoff sharp on this tower, vanishes on the neighboring one | UMAP at k=15 and k=50k will tell two different biologies |
| w flees to the extreme (w→0 or w→∞) | One metric (euclidean/cosine) assembles things that have no common p |
If that comes out, UMAP may be run as a QC screen (batch, obvious islands, junk outliers). But you must not read: distances between islands, a “trajectory” drawn with a pencil, similarity rankings from the layout, or “this marker pulls that one”.
If there is a repeatable trunk, stable LCP, Fisher with contrast in the middle of the scale, w around 1 or gently adapted - then UMAP gets what it silently requires, i.e. a locally assemblable neighborhood. Then the drawing is a projection of something that first passed the test. That something is, of course, the coherence of the database.
So if HMFoDG shows that something has no relation, and the researchers’ observational intuition says that it does, it means that something may have gone wrong with data collection - because we study data, not reality.
The test will not show that “the data are bad” (they are what they are). Only that at this resolution and at this x_0 there is no homogeneous assembly. Another horizon, another idiolect of markers, another foliation of conditions may show a trunk (this is in the previous article on snow on the screen).
HMFoDG does not replace UMAP. It does not produce a cell atlas, it does not prove that a correlation “does not exist in the world”. It proves that in these data, under this projection, the shared space is false. UMAP draws exactly that space. HMFoDG examines the coherence of the data. Whether there even exists such an abstract labyrinth in which, between chambers (tokens) deformed depending on the trajectory (context), any corridors - even strongly blurred ones - can be drawn. That is: whether relations exist in the data, and how sharp they are if at all.
UMAP cannot refuse to glue, and HMFoDG can: when short LCP, no trunk and Fisher without a maximum is a precise label “do not pack this into one metric and one k, or you will get forced bridges”. First the assemblability test, then if the test lets it through we make the drawing. Otherwise we get a picture of a drawing algorithm, not of the data.
HMFoDG indicates what to strike from the interpretation (and only afterwards, if you want, from the atlas). And not from the file!
Three layers, from soft to hard:
Leave it on the picture, but without distances; islands are visible, you do not read the bridges. QC, batches, coarse outliers.
Leave it, but foliate through another x_0, another prefix, another set of conditions. This is not one atlas, only several local maps. They may not connect, but locally each is fine for itself.
Do not assemble into one UMAP when LCP ~0, no trunk, smeared Fisher, w at the extreme. Here UMAP will invent correlations. Either you do not pack, or you pack separately along the trunks the audit distinguished.
The sanction is metric, not aesthetic: a common prefix and a stable w = you may read the neighborhood; no gluing of codes = an edge on UMAP is a decoration of the layout.
UMAP is a plotter of a local graph; HMFoDG is the stamp saying which edges of that graph are influence, and which are forced addressing of emptiness.
UMAP, PCA, cosine and “one L2 for the whole assay” silently assume a shared space. In biotech that assumption often falls because of mixed scales, empty volume, trajectories of conditions - and the plotter will still draw something. An audit that can refuse to glue then matters: it does not improve the coloring of the atlas, it says which edges may be taken into a conclusion.
The automaton in the loop is exactly what by hand is very expensive and in practice not executable with full effectiveness.
projection onto x_0 → R_w → sharpness (Fisher / entropy) → cutoff and w → prefix → does it glue → new x_0 / new data.
Without sitting over “which p to pick today”. The loop itself sets attenuation levels to the grain of the data and returns the metric of that calculation. This is still an assembly of known bricks (power means, graph, tree, Fisher) in one circuit, not a new ontology. What matters is that the circuit does not require anyone to know in advance which norm is true.
HMFoDG takes off the human a decision they could not justify anyway other than “we always did it this way”, and leaves them the proper decision: what from the sanctioned graph may be published as influence.
UPGMA is the closest cousin from the lab, not UMAP. UPGMA already returns a tree and HMFoDG also returns an ultrametric hierarchy - only not the same one and not from the same calculation.
Input: one distance matrix (usually one metric: Euclidean after z-score, correlation, Jaccard, p-distance on sequences).
Algorithm: glue pairs with the smallest average distance between clusters; node height = that average.
Output: dendrogram. Distance of leaves i,j is the height of the LCA and that is an ultrametric.
In phylogenetics the silent assumption is a clock: from the common ancestor both lineages “go” the same. Hence UPGMA breaks when the tempo is not equal (then neighbor-joining / ML). In gene expression nobody checks this because a tree appears anyway. Even though it is not in the data.
UPGMA cannot refuse. It glues every pair somewhere. The highest node always exists. Absence of a trunk comes out as a flat comb or as a junk trunk high up, not as “no common address”.
| UPGMA | HMFoDG | |
|---|---|---|
| Object | dendrogram from linkage | prefix tree, d∼p^{-LCP} or e^{-λ LCP} |
| NN | close at the height of the glue | NN ≡ max LCP (lemma from the source notes) |
The essential difference is where in HMFoDG height / lambda and the exponent w enter from local sharpness (Fisher, entropy, density). The average is not sacred. w > 1 glues by the dominant, w < 1 by the dispersed background, and UPGMA has no such switch.
UPGMA needs (or pretends the existence of) a global N×N matrix. Every pair has one number. HMFoDG: projection onto x_0, finite N(v). The same pair at another x_0 or another path has another measure. The path is part of the state, and in UPGMA the path in the tree is only the history of the algorithm’s gluings, not a trajectory of events.
This matters! A result in UPGMA can be a picture of the algorithm (see the problems of UMAP), not of the data.
UPGMA does not ask whether L2, correlation and Jaccard may be put into one matrix. Someone already decided that upstream. That is: a horse designed by a committee, humped, wandering the desert and living in an oasis. Still called a horse after the committee’s decision. This is of course a rant at a working assumption that a simple observation can overturn (the result from the data does not glue with observation in the experiment) - only the problem was what one was supposed to do about it.
HMFoDG is that upstream: which p, which cutoff. Only then the hierarchy. That is, instead of a committee, a calculation of which equivalence class of p will be closest to the data.
UPGMA assumes that the distance to the LCA is shared. Mixed biotech scales (dose, time, batch, rare event) are an anti-pattern of the clock.
HMFoDG, in the absence of a trunk, returns LCP close to 0, codes with no common position. That is “do not glue”, which linkage did not foresee.
UPGMA is insensitive to row order (at a fixed matrix).
In HMFoDG order / path / reprojection changes the address. For auditing embeddings and curriculum this is a feature, not a bug.
UPGMA: O(n²) on the matrix, done. HMFoDG: loop aggregation → scale scan → update prefixes → reprojection. More expensive intellectually, cheaper in the spirit of “do not compute the global emptiness”.
data / co-occurrences
│
▼
HMFoDG (is there a trunk? what w? do the codes glue?)
│
├─ no trunk / LCP~0 → do not pack into one UMAP nor into one UPGMA
│
└─ there is a trunk
│
├─ UMAP → picture of islands (without reading bridges)
└─ UPGMA → dendrogram *inside* the sanctioned trunk
UPGMA on raw distances of the whole atlas does the same thing UMAP does on a forced kNN: it invents an ancestor, because the algorithm must have a root.
UPGMA after HMFoDG is legal: we compute linkage only where the audit gave a common prefix and a stable w. This is classic “hierarchical clustering on a subset that is clusterable in the first place”.
Glossary:
| UPGMA | HMFoDG |
|---|---|
| height of LCA | length of the common prefix / λ⋅LCP |
| average between clusters | R_w, not necessarily w=1 |
| clock assumption | local sharpness; the clock is not an axiom |
| always a root | a root only when the codes glue |
| one distance matrix | local stars + flow |
| phylogenetic tree of expression | the address under which this expression is one hierarchy at all |
UPGMA is already an ultrametric that biotech knows how to read. HMFoDG is not “a better UPGMA”. It is a test of whether you may run UPGMA at all: first the trunk, w and cutoff, then a dendrogram on what glued. Without that, linkage - like UMAP - will always find a common ancestor, even when in the data there is only snow.
If You are intrested to dig deeper or just need someone to solve analitical assumptions in Your project....
The tool is implemented as a prototype tested for several months. If you are interested in the computational side then the entire loop of the following two equations (aggregation + generalized Fisher with free metric):
- Projection onto x0 → radial graph + v_i = x_i - x0.
- Local N(v), ordering y_i.
- Aggregator: R_w(y) = (∑ α_i |y_i|^w )^{1/w} [w adaptive, w=1 neutral].
- w < 1: diffuse/generalize
- w > 1: selective/amplify dominant
- Hierarchical addressing: directional/p-adic prefixes → ultrametric d = p^{-LCP} or e^{-λ LCP}.
- Lemma: NN ≡ max LCP → prefix tree = efficient hierarchy.
- Lambda/w from Fisher-like: local entropy H(N(v)), density, or spectral radius embeddability (ρ² < α).
- Flow: repeat aggregation → update metric/prefixes → reprojection with new data (redshift horizon).
Yields:
Advantages for ML/embeddings:
- Deterministic hierarchy (sharp taxonomies, noise control).
- Ordering of training instead of post-hoc censorship.
- Strong local metric deformation (tunnels, contrasts).
- Easily implementable (prefixes + LCP << cosine brute force).
This procedure is applied in the tool, from which I can list embeddings.
The computational and proof explanation of this loop is in:
- PAPERS_Hierarchical_Metric_Flow_on_Data_Graphs.txt (txt 175.5 kB) (main)
- PAPERS_appendix_implementations_1_for_Hierarchical_Metric_Flow_on_Data_Graphs.txt (txt 71 kB) (ultrametric + graph)
- PAPERS_HPF_QCO_tower_horizons.txt (txt 130 kB) (HPF + tower + spectral)
And curiosities are listed here:
- Math side: PAPERS_math_side_of_order_in_training_ENG.txt (txt 107.9 kB) (application to ML)
- Blogpost: HMFoDG Implementation - Analytical Curiosity for RAG Auditing.
Feel free to contact me if You have an analitical assumptions to solve.
-- Jack Kowalski