allotaxonometer-ui 0.1.20 → 0.2.1

package/README.md

@@ -1,13 +1,17 @@
 
+
 # Allotaxonometer UI
 
+
 <img width="50%" alt="pipeline" src="https://github.com/user-attachments/assets/41375b5f-e942-499f-aea4-bca004767e52">
 
-Headless UI components for allotaxonometer visualizations built with **Svelte 5**.
+<p>
 
 - Visit our [single-page web application](https://vermont-complex-systems.github.io/complex-stories/allotaxonometry) to try it out online.
-- Use [allotaxonometer-core](https://github.com/jstonge/allotaxonometer-core) for Rust/Python server-side computation.
-- See the original [py-allotax](https://github.com/compstorylab/py-allotax) for the Python-only implementation.
+- Use [py-allotax](https://github.com/compstorylab/py-allotax) to use allotaxonometry programmatically ([:octocat: github](https://github.com/compstorylab/py-allotax))
+- We also maintain a [matlab version](https://gitlab.com/compstorylab/allotaxonometer) to use allotaxonometry at scale.
+
+Headless UI components for allotaxonometer visualizations built with Svelte 5.
 
 ## Installation
 
@@ -15,112 +19,146 @@ Headless UI components for allotaxonometer visualizations built with **Svelte 5*
 npm install allotaxonometer-ui
 ```
 
-## Usage
+## Use cases and install
 
-### Client-side pipeline (Allotaxonograph class)
+- Explore systems and formulate research questions: we recommend starting with the web app...
+- Running multiple or larger scale analyses: we recommend using the py-allotax, which details its install instructions, details the required data format, and provides examples in its repo.
 
-Compute everything in the browser using the built-in JS/WASM pipeline:
-
-```svelte
-<script>
-  import { Allotaxonograph, Dashboard } from 'allotaxonometer-ui';
+## Data Input
 
-  const instance = new Allotaxonograph(elem1, elem2, {
-    alpha: 0.58,
-    title: ['System 1', 'System 2']
-  });
+The allotaxonometer expects 2 datasets with `types` and `counts`. The `totalunique` and `probs` fields are **optional** and will be calculated automatically if not provided.
 
-  let dat = $derived(instance?.dat);
-  let barData = $derived(instance?.barData);
-  let balanceData = $derived(instance?.balanceData);
-</script>
+### Minimal Format (Recommended)
+```javascript
+const data = [{
+  types: ['John', 'William', 'James', 'George', 'Charles'],
+  counts: [8502, 7494, 5097, 4458, 4061]
+  // totalunique and probs calculated automatically!
+}];
+```
 
-<Dashboard {dat} {barData} {balanceData}
-  divnorm={instance.rtd.normalization}
-  alpha={0.58}
-  maxlog10={instance.maxlog10}
-  title={['System 1', 'System 2']}
-/>
+### Full Format (All Fields)
+|    | types   |   counts |   totalunique |   probs |
+|----|---------|----------|---------------|---------|
+|  0 | John    |     8502 |          1161 |  0.0766 |
+|  1 | William |     7494 |          1161 |  0.0675 |
+|  2 | James   |     5097 |          1161 |  0.0459 |
+|  3 | George  |     4458 |          1161 |  0.0402 |
+|  4 | Charles |     4061 |          1161 |  0.0366 |
+
+
+## Paper data
+
+#### Babynames data
+
+The original babyname dataset for boys and girls can be found on the [catalog.data.gov](https://catalog.data.gov/dataset?tags=baby-names) website. But we use the dataset [here](http://pdodds.w3.uvm.edu/permanent-share/pocs-babynames.zip) to replicate the original paper. You can find a 5-years aggregated version used in the `Observable` version in `data/`. The original dataset includes each year from 1880–2018, which have 5 or more applications. You can convert the original folder into the formatted `.json` file using R with the following command:
+
+```R
+read_and_write_babyname_dat <- function(fname, gender) {
+  d <- readr::read_csv(fname, 
+                       col_names = c("types", "gender", "counts"), 
+                       col_select = c("types", "counts"),
+                       col_types = c("c", "i"))
+  
+  d$probs <- d$counts / sum(d$counts)
+  d$total_unique <- nrow(d)
+  return(d)
+}
+# You need to be in the folder above `data/`, which is the unzip folder contained in 
+# http://pdodds.w3.uvm.edu/permanent-share/pocs-babynames.zip
+purrr::map(
+  list.files("data/", pattern = "names-boys*"), 
+  ~read_and_write_babyname_dat(paste("data", .x, sep = "/"), "boys")
+)
+
+purrr::map(
+  list.files("data/", pattern = "names-girls*"), 
+  ~read_and_write_babyname_dat(paste("data", .x, sep = "/"), "girls")
+)
 ```
 
-### Server-computed data (API adapter)
+#### Twitter data
 
-When using a backend that computes allotax results (e.g. via `allotaxonometer-core`), use `adaptDisplayResult` to map the API response to component-ready props:
+We access the Twitter data from the Comptuational Story Lab [storywrangling](https://gitlab.com/compstorylab/storywrangling)' API. Unfortunately, the API only work when you are connected on the University of Vermont's VPN. Follow the instructions [here](https://www.uvm.edu/it/kb/article/install-cisco-vpn/) to get the VPN working. Once this is done, run the following lines from the command line:
 
-```js
-import { adaptDisplayResult } from 'allotaxonometer-ui';
+```shell
+git clone https://gitlab.com/compstorylab/storywrangling.git
+cd storywrangling
+pip install -e .
+```
 
-const apiResult = await fetch('/allotax?...').then(r => r.json());
-const display = adaptDisplayResult(apiResult);
-// display.dat, display.barData, display.balanceData, display.divnorm, display.maxlog10, etc.
+Then from `python` you can get the top ngram count with rank data for any given day with the following:
+
+```python
+from storywrangling import Storywrangler
+from datetime import datetime
+import json
+from pathlib import Path
+
+def get_ngram(yr, month, day, fname=False):
+    storywrangler = Storywrangler()
+    ngram_zipf = storywrangler.get_zipf_dist(
+        date=datetime(yr, month, day),
+        lang="en", ngrams="1grams",
+        max_rank=10000, rt=False
+    ).reset_index()\
+     .rename(columns={
+        "ngram":"types", "count":"counts", "count_no_rt":"counts_no_rt",
+        "rank":"rank", "rank_no_rt":"rank_no_rt", "freq":"probs", "freq_no_rt":"probs_no_rt"
+        })\
+     .dropna()\
+     .assign(totalunique = lambda x: x.shape[0])\
+     .loc[:, ["types", "counts", "totalunique", "probs"]]\
+     .to_dict(orient="index")
+
+    ngram_zipf = { f"{yr}_{month}_{day}": [_ for _ in ngram_zipf.values()] }
+
+    if fname:
+        if Path(fname).exists():
+            with open(fname) as f:
+                old_dat = json.load(f)
+            
+            ngram_zipf.update(old_dat)
+
+        with open(fname, 'w') as f:
+            json.dump(ngram_zipf, f)
+    else:
+       return ngram_zipf
 ```
 
-### Multi-alpha (recommended for API usage)
+Note that this solution is a bit clunky. At some point we would prefer to have a sql DB that we can interact with. 
 
-Fetch all alpha values at once and switch instantly on the client:
+#### Species Abundance Data
 
-```svelte
-<script>
-  // Fetch once with all alphas — no refetch when alpha slider changes
-  const data = await fetch('/allotax?alphas=0,0.08,0.17,...').then(r => r.json());
-  // data.balance — shared across alphas
-  // data.alpha_results[i] — { diamond_counts, wordshift, normalization, alpha }
-</script>
+We access the species abundance data from https://datadryad.org/stash/dataset/doi:10.15146/5xcp-0d46, downloading the full dataset, unzipping it, and then loading bci.tree\<i\>.rdata for i in (1-8), as well as bci.spptable.rdata. We then run the following code to subset the full census represented by each of the bci.tree\<i\>.rdata to get the counts of the species of the trees alive during that census, combine merge that with the species name database to get the full name, and then put it in the format that our allotaxonometer code expects:
 
-{@const slice = data.alpha_results[alphaIndex]}
-{@const dat = { counts: slice.diamond_counts, deltas: [] }}
+```r    
+library(Sys)
+library(dplyr)
+library("rlist")
+library(jsonlite)
 
-<Dashboard
-  {dat}
-  barData={slice.wordshift}
-  balanceData={data.balance}
-  divnorm={slice.normalization}
-  alpha={slice.alpha}
-/>
-```
+tree_data <- vector("list", length=8)
 
-## Components
 
-| Component | Description |
-|-----------|-------------|
-| `Dashboard` | Main orchestrating component combining all visualizations |
-| `Diamond` | Rank-rank scatter plot showing relationships between systems |
-| `Wordshift` | Horizontal bar chart showing type contributions to divergence |
-| `DivergingBarChart` | Bar chart for positive/negative shifts |
-| `Legend` | Color and size legends |
 
-## Data Input
+dfs = list(bci.tree1, bci.tree2, bci.tree3, bci.tree4,  bci.tree5, bci.tree6, bci.tree7, bci.tree8)
 
-The allotaxonometer expects 2 datasets with `types` and `counts`. The `totalunique` and `probs` fields are **optional** and will be calculated automatically if not provided.
 
-### Minimal Format (Recommended)
-```javascript
-const data = [{
-  types: ['John', 'William', 'James', 'George', 'Charles'],
-  counts: [8502, 7494, 5097, 4458, 4061]
-  // totalunique and probs calculated automatically!
-}];
-```
+for (i in seq_along(dfs)) {
+  print(i)
+  full_census <- merge(dfs[[i]], bci.spptable, by='sp') 
+  alive_census <-full_census[full_census$status %in% c('A','AD'),] # A='Alive', AD='A seldom-used code, applied when a tree was noted as dead in one census but was found alive in a later census. For most purposes, this code should be interpreted the same as code A for alive.'
+  count_df <- dplyr::count(alive_census, Latin, sort = TRUE)
+  names(count_df)[names(count_df) == 'Latin'] <- 'types'
+  names(count_df)[names(count_df) == 'n'] <- 'counts'
+  count_df['totalunique'] <- nrow(count_df)
+  count_df['probs']<-count_df['counts'] / nrow(alive_census)
+  tree_data[[i]] <- count_df
+}
 
-### Full Format
-```javascript
-const data = [{
-  types: ['John', 'William', 'James', 'George', 'Charles'],
-  counts: [8502, 7494, 5097, 4458, 4061],
-  totalunique: 1161,
-  probs: [0.0766, 0.0675, 0.0459, 0.0402, 0.0366]
-}];
-```
+names(tree_data) <- c("1981-1983", "1985", "1991-1992", "1995-1996", "2000-2001", "2005-2006", "2010-2011", "2013-2015")
 
-## Development
-
-```bash
-npm run build          # Build library (includes WASM)
-npm run build:wasm     # Build Rust/WASM module only
-npm run test:run       # Run all tests once
-npm test               # Run tests in watch mode
+exportJson <- toJSON(tree_data)
+write(exportJson, "tree_species_counts.json")
 ```
-
-## WASM Acceleration
-
-The `rank_turbulence_divergence` computation is implemented in Rust/WASM for 3-4x speedup on datasets with 10K+ types. WASM loads eagerly but falls back to JavaScript if unavailable. See [WASM_IMPLEMENTATION.md](WASM_IMPLEMENTATION.md) for details.