ro 4.2.0 → 4.4.0

data/public/api/ro/nerd/fastest-possible-embeddings/index.json

@@ -0,0 +1,90 @@
+{
+  "data": {
+    "nerd/fastest-possible-embeddings": {
+      "og": {
+        "title": "super fast embeddings with rust, javascript, ruby, ollama, and mistral.",
+        "description": "T.I.L how to generate fast embeddings locally with open source tools",
+        "image": {
+          "url": "/ro/nerd/fastest-possible-embeddings/assets/giraffe.jpeg",
+          "width": 2048
+        }
+      },
+      "body": "<div class=\"ro markdown\">\n  <h3 id=\"tldr\">TL;DR;</h3>\n\n<blockquote>\n  <p>today i figured out the fasest possible way to generate embeddings locally with open source tools avaiable today</p>\n</blockquote>\n\n<hr />\n<p><strong>update 2025-02-24</strong></p>\n\n<p>@kris (<a href=\"https://github.com/khasinski\" target=\"_blank\">https://github.com/khasinski</a>) helped me with this -&gt; <a href=\"https://gist.github.com/ahoward/2a1d45499ac9e755d802dbcbaf401b71\" target=\"_blank\">https://gist.github.com/ahoward/2a1d45499ac9e755d802dbcbaf401b71</a></p>\n\n<p>it.  is fast #af.</p>\n\n<hr />\n<p>i tried 4 approaches, based on <em>extensive</em> research and lots and lots of testing in my work from the past 8 mos.</p>\n\n<p>i’ve been busy building ‘socially informed AI’.   essentially, a giant,\ncurated, enahanced, and distilled RAG database of over a million conversations\nacross a thousand subreddits, domains, etc.  it’s a vast database of\na million conversations that can be brought to bear for literally <strong>ANY</strong> task\none might do with <strong>AI</strong>.</p>\n\n<p>i have a beta program now.  ping me directly if you want an <strong>API</strong> key.  it\nwill be developer-first in the immediate future.  it’s a sharp tool i don’t\nwant used for the wrong purposes at this juncture.   a self serve product is\nin the works… more on that soon.</p>\n\n<p>anyhow, let’s say i need to generate embeddings for all these conversations.\nwhat <em>is</em> the fastest way?</p>\n\n<p>i tried 4 methods:</p>\n\n<ol>\n  <li>ruby wrapper around <a href=\"https://mistral.ai/\" target=\"_blank\">https://mistral.ai/</a></li>\n  <li>ruby wrapper on <a href=\"https://ollama.com/\" target=\"_blank\">https://ollama.com/</a></li>\n  <li>rust command line program <a href=\"https://github.com/Anush008/fastembed-rs\" target=\"_blank\">https://github.com/Anush008/fastembed-rs</a></li>\n  <li>node command line programm using <a href=\"https://github.com/Anush008/fastembed-js\" target=\"_blank\">https://github.com/Anush008/fastembed-js</a></li>\n</ol>\n\n<h5 id=\"here-is-the-ruby-code-to-generate-embeddings-with-mistral--very-short--very-easy--i-expected-this-to-be-the-slowest-although-it-took-me-only-5-minutes-to-write\">here is the ruby code to generate embeddings with mistral.  very short.  very easy.  i expected this, to be the slowest, although it took me only 5 minutes to write.</h5>\n\n<div class=\"language-ruby highlighter-rouge\"><code class='code' style='white-space:pre;'><div class='code-line code-line-1'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">#! /usr/bin/env ruby</span>\n</div><div class='code-line code-line-2'>\n</div><div class='code-line code-line-3'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">require</span>&nbsp;<span style=\"color: #0a3069\">'mistral-ai'</span>\n</div><div class='code-line code-line-4'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">require</span>&nbsp;<span style=\"color: #0a3069\">'json'</span>\n</div><div class='code-line code-line-5'>\n</div><div class='code-line code-line-6'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">input</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-7'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">ARGV</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">shift</span>\n</div><div class='code-line code-line-8'>\n</div><div class='code-line code-line-9'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">prompt</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-10'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">input</span>&nbsp;<span style=\"color: #0550ae\">==</span>&nbsp;<span style=\"color: #0a3069\">'-'</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">?</span>&nbsp;<span style=\"color: #953800\">STDIN</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">read</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">:</span>&nbsp;<span style=\"color: #953800\">IO</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">binread</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">file</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>\n</div><div class='code-line code-line-11'>\n</div><div class='code-line code-line-12'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">mistral</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-13'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">Mistral</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">new</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span>\n</div><div class='code-line code-line-14'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #0a3069\">credentials: </span><span style=\"color: #24292f;background-color: #f6f8fa\">{</span>&nbsp;<span style=\"color: #0a3069\">api_key: </span><span style=\"color: #953800\">ENV</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">fetch</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">'MISTRAL_API_KEY'</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">},</span>\n</div><div class='code-line code-line-15'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #0a3069\">options: </span><span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-16'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #0a3069\">connection: </span><span style=\"color: #24292f;background-color: #f6f8fa\">{</span>&nbsp;<span style=\"color: #0a3069\">request: </span><span style=\"color: #24292f;background-color: #f6f8fa\">{</span>&nbsp;<span style=\"color: #0a3069\">timeout: </span><span style=\"color: #0550ae\">420</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">},</span>\n</div><div class='code-line code-line-17'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-18'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">)</span>\n</div><div class='code-line code-line-19'>\n</div><div class='code-line code-line-20'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">model</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-21'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #0a3069\">'mistral-embed'</span>\n</div><div class='code-line code-line-22'>\n</div><div class='code-line code-line-23'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">result</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-24'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">mistral</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">embeddings</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span>\n</div><div class='code-line code-line-25'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>&nbsp;<span style=\"color: #0a3069\">model: </span><span style=\"color: #24292f;background-color: #f6f8fa\">model</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>\n</div><div class='code-line code-line-26'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #0a3069\">input: </span><span style=\"color: #24292f;background-color: #f6f8fa\">inputs</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>\n</div><div class='code-line code-line-27'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-28'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">)</span>\n</div><div class='code-line code-line-29'>\n</div><div class='code-line code-line-30'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embedding</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-31'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">result</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">fetch</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">:data</span><span style=\"color: #24292f;background-color: #f6f8fa\">).</span><span style=\"color: #8250df\">fetch</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">0</span><span style=\"color: #24292f;background-color: #f6f8fa\">).</span><span style=\"color: #8250df\">fetch</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">:embedding</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>\n</div><div class='code-line code-line-32'>\n</div><div class='code-line code-line-33'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">json</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-34'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">JSON</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">generate</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">embedding: </span><span style=\"color: #24292f;background-color: #f6f8fa\">embedding</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>\n</div><div class='code-line code-line-35'>\n</div><div class='code-line code-line-36'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">puts</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">json</span>\n</div></code></div>\n<p><br /></p>\n\n<h5 id=\"next-i-installed-httpsollamacom-and-wrote-this-silly-ruby-wrapper-to-use-it--bout-3-minutes-to-write\">next, i installed <a href=\"https://ollama.com/\" target=\"_blank\">https://ollama.com/</a> and wrote this silly ruby wrapper to use it.  bout 3 minutes to write.</h5>\n\n<div class=\"language-ruby highlighter-rouge\"><code class='code' style='white-space:pre;'><div class='code-line code-line-1'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">#! /usr/bin/env ruby</span>\n</div><div class='code-line code-line-2'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">require</span>&nbsp;<span style=\"color: #0a3069\">'json'</span>\n</div><div class='code-line code-line-3'>\n</div><div class='code-line code-line-4'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">input</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-5'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">ARGV</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">shift</span>\n</div><div class='code-line code-line-6'>\n</div><div class='code-line code-line-7'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">prompt</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-8'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">input</span>&nbsp;<span style=\"color: #0550ae\">==</span>&nbsp;<span style=\"color: #0a3069\">'-'</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">?</span>&nbsp;<span style=\"color: #953800\">STDIN</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">read</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">:</span>&nbsp;<span style=\"color: #953800\">IO</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">binread</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">file</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>\n</div><div class='code-line code-line-9'>\n</div><div class='code-line code-line-10'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">model</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-11'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #0a3069\">\"mxbai-embed-large\"</span>\n</div><div class='code-line code-line-12'>\n</div><div class='code-line code-line-13'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">data</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-14'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span><span style=\"color: #24292f;background-color: #f6f8fa\">prompt</span><span style=\"color: #24292f;background-color: #f6f8fa\">:,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">model</span><span style=\"color: #24292f;background-color: #f6f8fa\">:,}</span>\n</div><div class='code-line code-line-15'>\n</div><div class='code-line code-line-16'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">cmd</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-17'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #0a3069\">\"curl -s 'http://localhost:11434/api/embeddings' -d '</span><span style=\"color: #24292f\">#{</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">data</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">to_json</span>&nbsp;<span style=\"color: #24292f\">}</span><span style=\"color: #0a3069\">'\"</span>\n</div><div class='code-line code-line-18'>\n</div><div class='code-line code-line-19'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">json</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-20'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">IO</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">popen</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">cmd</span><span style=\"color: #24292f;background-color: #f6f8fa\">).</span><span style=\"color: #8250df\">read</span>\n</div><div class='code-line code-line-21'>\n</div><div class='code-line code-line-22'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">puts</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">json</span>\n</div></code></div>\n<p><br /></p>\n\n<h5 id=\"finally-i-wrote-a-rust-and-nodejavascript-version-see-below-which-were-much-longer-and-took-a-little-time-to-produce-even-with-ai-codebots-vs-code-and-a-pile-of-shit-to-install\">finally, i wrote a rust and node/javascript version (see below) which were much longer and took a little time to produce, even <strong>with</strong> ai codebots, vs-code, and a pile of shit to install.</h5>\n\n<p>the numbers.  roughly, for a 4 page document, on my linux lappy, which is pretty dang beefy but <strong>not a GPU</strong></p>\n\n<pre><code class=\"language-code\">drawohara@drawohara.dev:disco[main] #=&gt; cat /proc/cpuinfo | mistral-ai\n\nThe information you provided is the output from a command like `lscpu` or `cat /proc/cpuinfo` on a Linux system, detailing the specifications of each processor core in a multi-core AMD EPYC 7713 64-Core Processor. Here's a breakdown of the key details:\n\n### General Processor Information\n- **Vendor ID**: AuthenticAMD\n- **Model Name**: AMD EPYC 7713 64-Core Processor\n- **CPU Family**: 25\n- **Model**: 1\n- **Stepping**: 1\n- **Microcode**: 0xa0011d1\n- **CPU MHz**: 1999.999 MHz (2 GHz base frequency)\n- **Cache Size**: 512 KB (L2 cache size per core)\n- **Physical ID**: 0 (Indicates it is part of the same physical package)\n- **Siblings**: 4 (Indicates the number of logical processors per core, due to hyper-threading)\n- **CPU Cores**: 4 (Number of cores reported by this logical processor)\n\n### Core-Specific Information\n- **Core ID**: Ranges from 0 to 3 (Indicates the specific core within the physical package)\n- **APIC ID**: Ranges from 0 to 3 (Advanced Programmable Interrupt Controller ID)\n- **Initial APIC ID**: Ranges from 0 to 3\n\n### CPU Features\n- **Flags**:\n  - Standard features: `fpu`, `vme`, `de`, `pse`, `tsc`, `msr`, `pae`, `mce`, `cx8`, `apic`, `sep`, `mtrr`, `pge`, `mca`, `cmov`, `pat`, `pse36`, `clflush`, `mmx`, `fxsr`, `sse`, `sse2`, `ss`, `ht`, `syscall`, `nx`, `pdpe1gb`, `rdtscp`, `lm`, `constant_tsc`, `rep_good`, `nopl`, `xtopology`, `nonstop_tsc`, `aperf`, `eagerfpu`, `pni`, `pclmulqdq`, `dtes64`, `monitor`, `ds_cpl`, `vmx`, `est`, `tm2`, `ssse3`, `cx16`, `xtpr`, `pdcm`, `pcid`, `sse4_1`, `sse4_2`, `x2apic`, `movbe`, `popcnt`, `tsc_deadline_timer`, `aes`, `xsave`, `avx`, `f16c`, `rdrand`, `lahf_lm`, `cmp_legacy`, `cr8_legacy`, `abm`, `sse4a`, `misalignsse`, `3dnowprefetch`, `osvw`, `skinit`, `wdt`, `nodeid_msr`, `topoext`, `perfctr_core`, `perfctr_nb`, `bpext`, `ptwrite`, `mwaitx`, `cat_l3`, `cat_l2`, `arat`, `pln`, `mce_am`, `powercnt`, `ssbd`, `ibrs`, `ibpb`, `stibp`, `vmmcall`, `fsgsbase`, `bmi1`, `hle`, `avx2`, `smep`, `bmi2`, `erms`, `invpcid`, `rtm`, `cqm`, `mpx`, `rdseed`, `adx`, `smap`, `clflushopt`, `clwb`, `sha_ni`, `xsaveopt`, `xsavec`, `xgetbv1`, `xsaves`, `avx512f`, `avx512dq`, `avx512pf`, `avx512er`, `avx512cd`, `avx512bw`, `avx512vl`, `prefetchwt1`, `clzero`, `avx512ifma`, `avx512vbmi`, `umip`, `pku`, `ospke`, `waitpkg`, `avx512_vnni`, `avx512bitalg`, `rdpid`, `fsrm`, `vaes`, `vpclmulqdq`, `arch_capabilities`\n- **Bugs**: `sysret_ss_attrs`, `null_seg`, `spectre_v1`, `spectre_v2`, `spec_store_bypass`, `srso`\n\n### Performance and Cache Information\n- **BogoMIPS**: 3999.99 (A relative measure of processor speed)\n- **TLB Size**: 1024 4K pages\n- **Clflush Size**: 64 bytes\n- **Cache Alignment**: 64 bytes\n- **Address Sizes**: 48 bits physical, 48 bits virtual\n\n### Power Management\n- The output does not provide specific details on power management features, but modern CPUs like the EPYC 7713 will have various power management capabilities.\n\n### Summary\nThe AMD EPYC 7713 is a high-performance server processor with 64 cores and advanced features such as SMT (Simultaneous Multi-Threading), large cache sizes, and support for a wide range of instruction sets and security features. The output shows that each physical core has 4 logical processors due to hyper-threading, and all cores are part of the same physical package. The processor operates at a base frequency of 2 GHz and includes numerous features and capabilities suitable for enterprise-level computations.\n</code></pre>\n<p><br /></p>\n\n<h5 id=\"the-results-really-surprised-me\">the results really surprised me</h5>\n\n<ol>\n  <li>ruby + remote mistral api call = ~ 1.75 seconds per embedding</li>\n  <li>ruby + local ollama api call = ~ 1.5 seconds per embedding</li>\n  <li>javascript + fastembed-js = ~ 0.7 seconds per embedding</li>\n  <li>rust + fastembed-rs = ~ 0.3 seconds per embedding</li>\n</ol>\n\n<p>so, here is my math:</p>\n\n<div class=\"language-ruby highlighter-rouge\"><code class='code' style='white-space:pre;'><div class='code-line code-line-1'>\n</div><div class='code-line code-line-2'><span style=\"color: #953800\">NUMBER_CONVERSATIONS_IN_DISCO</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-3'>&nbsp;&nbsp;<span style=\"color: #0550ae\">1_000_000</span>\n</div><div class='code-line code-line-4'>\n</div><div class='code-line code-line-5'><span style=\"color: #953800\">SECONDS_PER_CONVERSATION</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-6'>&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">0.3</span>&nbsp;<span style=\"color: #0550ae\">..</span>&nbsp;<span style=\"color: #0550ae\">1.7</span><span style=\"color: #24292f;background-color: #f6f8fa\">).</span><span style=\"color: #8250df\">to_a</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">sample</span>\n</div><div class='code-line code-line-7'>\n</div><div class='code-line code-line-8'><span style=\"color: #953800\">TOTAL_SECONDS</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-9'>&nbsp;&nbsp;<span style=\"color: #953800\">NUMBER_CONVERSATIONS_IN_DISCO</span>&nbsp;<span style=\"color: #0550ae\">*</span>&nbsp;<span style=\"color: #953800\">SECONDS_PER_CONVERSATION</span>\n</div><div class='code-line code-line-10'>\n</div><div class='code-line code-line-11'><span style=\"color: #953800\">TOTAL_MINUTES</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-12'>&nbsp;&nbsp;<span style=\"color: #953800\">TOTAL_SECONDS</span>&nbsp;<span style=\"color: #0550ae\">/</span>&nbsp;<span style=\"color: #0550ae\">60</span>\n</div><div class='code-line code-line-13'>\n</div><div class='code-line code-line-14'><span style=\"color: #953800\">TOTAL_HOURS</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-15'>&nbsp;&nbsp;<span style=\"color: #953800\">TOTAL_MINUTES</span>&nbsp;<span style=\"color: #0550ae\">/</span>&nbsp;<span style=\"color: #0550ae\">60</span>\n</div><div class='code-line code-line-16'>\n</div><div class='code-line code-line-17'><span style=\"color: #953800\">TOTAL_DAYS</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-18'>&nbsp;&nbsp;<span style=\"color: #953800\">TOTAL_HOURS</span>&nbsp;<span style=\"color: #0550ae\">/</span>&nbsp;<span style=\"color: #0550ae\">24</span>\n</div><div class='code-line code-line-19'>\n</div></code></div>\n<p><br /></p>\n\n<p>basically, with rust + fastembed-js (the fastest): <strong>about 4 days run-time to do all the embeddings</strong></p>\n\n<p>with ruby + mistral-ai (the slowest): <strong>about 20 days run-time to do all the embeddings</strong></p>\n\n<p>so, rust is a clear winner yes?  despite a whole tool chain of depenencies needed, compilation steps, integration into ci, etc.?</p>\n\n<blockquote>\n  <h5 id=\"not-so-fast\">NOT SO FAST</h5>\n</blockquote>\n\n<p>remember, what i showed about my machine.  for each of the approaches that run\nlocally, include ollama, rust, and javascript, i am</p>\n\n<p><strong>limited by how many cpus i have</strong></p>\n\n<p>through testing, i have found that running about 4 at a time gives the fastest throughput.  ymmv.</p>\n\n<p>whereas, with mistral, i can run using the <code>parallel</code> gem and process <strong>20 at a time with mistral</strong>, due to API limits. and this is <em>stunningly</em> easy:</p>\n\n<div class=\"language-ruby highlighter-rouge\"><code class='code' style='white-space:pre;'><div class='code-line code-line-1'>\n</div><div class='code-line code-line-2'>&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">prompts</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-3'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">[]</span>\n</div><div class='code-line code-line-4'>\n</div><div class='code-line code-line-5'>&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">rps</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-6'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #0550ae\">20</span>\n</div><div class='code-line code-line-7'>\n</div><div class='code-line code-line-8'>&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embeddings</span>&nbsp;<span style=\"color: #0550ae\">=</span>\n</div><div class='code-line code-line-9'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">Parallel</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">map</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">prompts</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #0a3069\">in_processes: </span><span style=\"color: #24292f;background-color: #f6f8fa\">rps</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #cf222e\">do</span>&nbsp;<span style=\"color: #0550ae\">|</span><span style=\"color: #24292f;background-color: #f6f8fa\">prompt</span><span style=\"color: #0550ae\">|</span>\n</div><div class='code-line code-line-10'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embedding_for</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">prompt</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>\n</div><div class='code-line code-line-11'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">end</span>\n</div><div class='code-line code-line-12'>\n</div></code></div>\n<p><br /></p>\n\n<p>yes, i <strong>do actually have a rate_limter</strong> but, since they take &gt; 1 second each that simple code actually never hits the limit ;-)</p>\n\n<p>finally, here is <strong>the main thing</strong></p>\n\n<blockquote>\n  <p>4 days / 4</p>\n</blockquote>\n\n<p><strong>==</strong></p>\n\n<blockquote>\n  <p>20 days / 20</p>\n</blockquote>\n\n<p>because math.</p>\n\n<p>and, actually, with batching i can make the mistral version quite a bit faster, about 10x.</p>\n\n<p>so the moral is this</p>\n\n<blockquote>\n  <p>the simplest and easiest solution was the best.</p>\n</blockquote>\n\n<p>and</p>\n\n<blockquote>\n  <p>premature optimization is the root of all evil.</p>\n</blockquote>\n\n<p>someone <a href=\"https://wiki.c2.com/?PrematureOptimization\">very clever once said that^^^^^!</a></p>\n\n<p>in nearly 30 years doing both big, and i mean <a href=\"/nerd\">really big</a> data and science with ruby, i have yet to meet a better <a href=\"https://en.wikipedia.org/wiki/Very_high-level_programming_language\">VHLL</a> to model my abstractions in, and to get shit done <strong>FAST</strong></p>\n\n<p>for sure, i occasionally will need to drop into rust or c but, those cases, are more rare than you might think and, often, reaching for the ‘big guns’ too fast just wastes time.</p>\n\n<p>certainly, anything in-between a VHLL and a compiled lang is a waste of time,\nmoney, and effort.  here’s looking at you javascript and pythong (ssss 🐍)!  haha ;-)</p>\n\n<p>ps.  because i can hear you thinking</p>\n\n<blockquote>\n  <p>i can call APIs in pythong (🐍🐍🐍🐍sssss) and js too</p>\n</blockquote>\n\n<p>and, of course, this is true.   especially if you enjoy worst-in-history\ndependency management, security issues, version churn, and really really like\nwriting 10x the LOC to accomplish the same damn thing that still blow up with\nhorrible stack traces of parallism in production your dev-ops team (fire them)\nstill can’t figure out no matter how many parameters they tune.</p>\n\n<p>next article is about dependency <strong>hell</strong> in pythong.  the time i worked at a\n“really great firm” that was “expert in python” (ssssss 🐍🐍🐍🐍🐍) and showed\nthem that, depsite 4 months of trying by thier principals, they could not\nmanage to have fewer than 4 versions of python running in CI and <strong>no one\neven knew</strong>.  but i will save that rant, for another day.</p>\n\n</div>\n",
+      "assets": {
+        "assets/giraffe.jpeg": {
+          "url": "/ro/nerd/fastest-possible-embeddings/assets/giraffe.jpeg",
+          "path": "nerd/fastest-possible-embeddings/assets/giraffe.jpeg",
+          "size": 8,
+          "img": {
+            "format": "jpeg",
+            "width": 2048,
+            "height": 2048
+          },
+          "src": null
+        },
+        "assets/let-me-in.jpg": {
+          "url": "/ro/nerd/fastest-possible-embeddings/assets/let-me-in.jpg",
+          "path": "nerd/fastest-possible-embeddings/assets/let-me-in.jpg",
+          "size": 8,
+          "img": {
+            "format": "jpeg",
+            "width": 1920,
+            "height": 1200
+          },
+          "src": null
+        },
+        "assets/terminal.jpg": {
+          "url": "/ro/nerd/fastest-possible-embeddings/assets/terminal.jpg",
+          "path": "nerd/fastest-possible-embeddings/assets/terminal.jpg",
+          "size": 8,
+          "img": {
+            "format": "jpeg",
+            "width": 1201,
+            "height": 1194
+          },
+          "src": null
+        },
+        "assets/src/fastembed.js": {
+          "url": "/ro/nerd/fastest-possible-embeddings/assets/src/fastembed.js",
+          "path": "nerd/fastest-possible-embeddings/assets/src/fastembed.js",
+          "size": 8,
+          "img": null,
+          "src": "<div class=\"ro markdown src\">\n  <code class='code' style='white-space:pre;'><div class='code-line code-line-1'><span style=\"color: #6e7781\">#! /usr/bin/env node</span>\n</div><div class='code-line code-line-2'>\n</div><div class='code-line code-line-3'><span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">fs</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #8250df\">require</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">'</span><span style=\"color: #0a3069\">fs</span><span style=\"color: #0a3069\">'</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-4'><span style=\"color: #6e7781\">//const { FastEmbed } = require('fastembed');</span>\n</div><div class='code-line code-line-5'><span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">EmbeddingModel</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">FlagEmbedding</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #8250df\">require</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">'</span><span style=\"color: #0a3069\">fastembed</span><span style=\"color: #0a3069\">'</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-6'>\n</div><div class='code-line code-line-7'><span style=\"color: #6e7781\">// Function to read input file</span>\n</div><div class='code-line code-line-8'><span style=\"color: #cf222e\">function</span>&nbsp;<span style=\"color: #8250df\">readInputFile</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">inputPath</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-9'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">if </span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">inputPath</span>&nbsp;<span style=\"color: #0550ae\">===</span>&nbsp;<span style=\"color: #0a3069\">'</span><span style=\"color: #0a3069\">-</span><span style=\"color: #0a3069\">'</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-10'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">return</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">fs</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">readFileSync</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">'</span><span style=\"color: #0a3069\">/dev/stdin</span><span style=\"color: #0a3069\">'</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #0a3069\">'</span><span style=\"color: #0a3069\">utf8</span><span style=\"color: #0a3069\">'</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-11'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>&nbsp;<span style=\"color: #cf222e\">else</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-12'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">return</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">fs</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">readFileSync</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">inputPath</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #0a3069\">'</span><span style=\"color: #0a3069\">utf8</span><span style=\"color: #0a3069\">'</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-13'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-14'><span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-15'>\n</div><div class='code-line code-line-16'><span style=\"color: #6e7781\">// Function to write output file</span>\n</div><div class='code-line code-line-17'><span style=\"color: #cf222e\">function</span>&nbsp;<span style=\"color: #8250df\">writeOutputFile</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">outputPath</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">data</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-18'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">if </span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">outputPath</span>&nbsp;<span style=\"color: #0550ae\">===</span>&nbsp;<span style=\"color: #0a3069\">'</span><span style=\"color: #0a3069\">-</span><span style=\"color: #0a3069\">'</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-19'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">process</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #24292f;background-color: #f6f8fa\">stdout</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">write</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">data</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-20'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>&nbsp;<span style=\"color: #cf222e\">else</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-21'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">fs</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">writeFileSync</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">outputPath</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">data</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #0a3069\">'</span><span style=\"color: #0a3069\">utf8</span><span style=\"color: #0a3069\">'</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-22'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-23'><span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-24'>\n</div><div class='code-line code-line-25'><span style=\"color: #6e7781\">// Main function</span>\n</div><div class='code-line code-line-26'><span style=\"color: #cf222e\">async</span>&nbsp;<span style=\"color: #cf222e\">function</span>&nbsp;<span style=\"color: #8250df\">main</span><span style=\"color: #24292f;background-color: #f6f8fa\">()</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-27'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">args</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">process</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #24292f;background-color: #f6f8fa\">argv</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">slice</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">2</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-28'>\n</div><div class='code-line code-line-29'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">if </span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">args</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #24292f;background-color: #f6f8fa\">length</span>&nbsp;<span style=\"color: #0550ae\">!==</span>&nbsp;<span style=\"color: #0550ae\">2</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-30'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">console</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">error</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">'</span><span style=\"color: #0a3069\">Usage: node program.js &lt;input&gt; &lt;output&gt;</span><span style=\"color: #0a3069\">'</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-31'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">process</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">exit</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">42</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-32'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-33'>\n</div><div class='code-line code-line-34'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">[</span><span style=\"color: #24292f;background-color: #f6f8fa\">inputPath</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">outputPath</span><span style=\"color: #24292f;background-color: #f6f8fa\">]</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">args</span><span style=\"color: #24292f;background-color: #f6f8fa\">;</span>\n</div><div class='code-line code-line-35'>\n</div><div class='code-line code-line-36'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">try</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-37'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">inputText</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #8250df\">readInputFile</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">inputPath</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-38'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//const embedder = new FastEmbed();</span>\n</div><div class='code-line code-line-39'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embedder</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #cf222e\">await</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">FlagEmbedding</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">init</span><span style=\"color: #24292f;background-color: #f6f8fa\">({</span>\n</div><div class='code-line code-line-40'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #116329\">model</span><span style=\"color: #24292f;background-color: #f6f8fa\">:</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">EmbeddingModel</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #24292f;background-color: #f6f8fa\">BGEBaseEN</span>\n</div><div class='code-line code-line-41'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">});</span>\n</div><div class='code-line code-line-42'>\n</div><div class='code-line code-line-43'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">documents</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">[</span><span style=\"color: #24292f;background-color: #f6f8fa\">inputText</span><span style=\"color: #24292f;background-color: #f6f8fa\">];</span>\n</div><div class='code-line code-line-44'>\n</div><div class='code-line code-line-45'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embeddings</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #cf222e\">await</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embedder</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">embed</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">documents</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-46'>\n</div><div class='code-line code-line-47'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">for</span>&nbsp;<span style=\"color: #cf222e\">await </span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">batch</span>&nbsp;<span style=\"color: #cf222e\">of</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embeddings</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-48'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">// batch is list of Float32 embeddings(number[][]) with length 2</span>\n</div><div class='code-line code-line-49'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//console.dir(batch);</span>\n</div><div class='code-line code-line-50'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embedding</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">batch</span><span style=\"color: #24292f;background-color: #f6f8fa\">[</span><span style=\"color: #0550ae\">0</span><span style=\"color: #24292f;background-color: #f6f8fa\">];</span>\n</div><div class='code-line code-line-51'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">const</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embeddingString</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #0a3069\">\"</span><span style=\"color: #0a3069\">[</span><span style=\"color: #0a3069\">\\n</span><span style=\"color: #0a3069\">\"</span>&nbsp;<span style=\"color: #0550ae\">+</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embedding</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">join</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">\"</span><span style=\"color: #0a3069\">,</span><span style=\"color: #0a3069\">\\n</span><span style=\"color: #0a3069\">\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #0550ae\">+</span>&nbsp;<span style=\"color: #0a3069\">\"</span><span style=\"color: #0a3069\">\\n</span><span style=\"color: #0a3069\">]</span><span style=\"color: #0a3069\">\\n</span><span style=\"color: #0a3069\">\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">;</span>\n</div><div class='code-line code-line-52'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//const embeddingString = JSON.stringify(embedding);</span>\n</div><div class='code-line code-line-53'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">writeOutputFile</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">outputPath</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embeddingString</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-54'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">process</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">exit</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">0</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-55'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">};</span>\n</div><div class='code-line code-line-56'>\n</div><div class='code-line code-line-57'><span style=\"color: #6e7781\">//debugger;</span>\n</div><div class='code-line code-line-58'><span style=\"color: #6e7781\">//console.dir(embedding);</span>\n</div><div class='code-line code-line-59'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//const embeddingString = embedding.join(',');</span>\n</div><div class='code-line code-line-60'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//writeOutputFile(outputPath, embeddingString);</span>\n</div><div class='code-line code-line-61'>\n</div><div class='code-line code-line-62'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//process.exit(0);</span>\n</div><div class='code-line code-line-63'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>&nbsp;<span style=\"color: #cf222e\">catch </span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">error</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-64'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">console</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">error</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">'</span><span style=\"color: #0a3069\">Error:</span><span style=\"color: #0a3069\">'</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">error</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #24292f;background-color: #f6f8fa\">message</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-65'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">process</span><span style=\"color: #24292f;background-color: #f6f8fa\">.</span><span style=\"color: #8250df\">exit</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">1</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-66'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-67'><span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-68'>\n</div><div class='code-line code-line-69'><span style=\"color: #8250df\">main</span><span style=\"color: #24292f;background-color: #f6f8fa\">();</span>\n</div><div class='code-line code-line-70'>\n</div></code>\n</div>\n"
+        },
+        "assets/src/fastembed.rs": {
+          "url": "/ro/nerd/fastest-possible-embeddings/assets/src/fastembed.rs",
+          "path": "nerd/fastest-possible-embeddings/assets/src/fastembed.rs",
+          "size": 8,
+          "img": null,
+          "src": "<div class=\"ro markdown src\">\n  <code class='code' style='white-space:pre;'><div class='code-line code-line-1'><span style=\"color: #cf222e\">use</span>&nbsp;<span style=\"color: #953800\">fastembed</span><span style=\"color: #24292f;background-color: #f6f8fa\">::{</span><span style=\"color: #24292f;background-color: #f6f8fa\">TextEmbedding</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">InitOptions</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">EmbeddingModel</span><span style=\"color: #24292f;background-color: #f6f8fa\">};</span>\n</div><div class='code-line code-line-2'><span style=\"color: #cf222e\">use</span>&nbsp;<span style=\"color: #953800\">serde</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #24292f;background-color: #f6f8fa\">Serialize</span><span style=\"color: #24292f;background-color: #f6f8fa\">;</span>\n</div><div class='code-line code-line-3'><span style=\"color: #cf222e\">use</span>&nbsp;<span style=\"color: #953800\">std</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #24292f;background-color: #f6f8fa\">env</span><span style=\"color: #24292f;background-color: #f6f8fa\">;</span>\n</div><div class='code-line code-line-4'><span style=\"color: #cf222e\">use</span>&nbsp;<span style=\"color: #953800\">std</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #24292f;background-color: #f6f8fa\">fs</span><span style=\"color: #24292f;background-color: #f6f8fa\">;</span>\n</div><div class='code-line code-line-5'>\n</div><div class='code-line code-line-6'><span style=\"color: #8250df\">#[derive(Serialize)]</span>\n</div><div class='code-line code-line-7'><span style=\"color: #cf222e\">struct</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">EmbeddingOutput</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-8'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embedding</span><span style=\"color: #24292f;background-color: #f6f8fa\">:</span>&nbsp;<span style=\"color: #953800\">String</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>\n</div><div class='code-line code-line-9'><span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-10'>\n</div><div class='code-line code-line-11'><span style=\"color: #cf222e\">fn</span>&nbsp;<span style=\"color: #8250df\">main</span><span style=\"color: #24292f;background-color: #f6f8fa\">()</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-12'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">args</span><span style=\"color: #24292f;background-color: #f6f8fa\">:</span>&nbsp;<span style=\"color: #953800\">Vec</span><span style=\"color: #0550ae\">&lt;</span><span style=\"color: #953800\">String</span><span style=\"color: #0550ae\">&gt;</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #953800\">env</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #8250df\">args</span><span style=\"color: #24292f;background-color: #f6f8fa\">()</span><span style=\"color: #8250df\">.collect</span><span style=\"color: #24292f;background-color: #f6f8fa\">();</span>\n</div><div class='code-line code-line-13'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">if</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">args</span><span style=\"color: #8250df\">.len</span><span style=\"color: #24292f;background-color: #f6f8fa\">()</span>&nbsp;<span style=\"color: #0550ae\">!=</span>&nbsp;<span style=\"color: #0550ae\">3</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-14'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">eprintln!</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">\"Usage: {} &lt;input_file&gt; &lt;output_file&gt;\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">args</span><span style=\"color: #24292f;background-color: #f6f8fa\">[</span><span style=\"color: #0550ae\">0</span><span style=\"color: #24292f;background-color: #f6f8fa\">]);</span>\n</div><div class='code-line code-line-15'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">std</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #953800\">process</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #8250df\">exit</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">1</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-16'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-17'>\n</div><div class='code-line code-line-18'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">input_file</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #0550ae\">&amp;</span><span style=\"color: #24292f;background-color: #f6f8fa\">args</span><span style=\"color: #24292f;background-color: #f6f8fa\">[</span><span style=\"color: #0550ae\">1</span><span style=\"color: #24292f;background-color: #f6f8fa\">];</span>\n</div><div class='code-line code-line-19'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">output_file</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #0550ae\">&amp;</span><span style=\"color: #24292f;background-color: #f6f8fa\">args</span><span style=\"color: #24292f;background-color: #f6f8fa\">[</span><span style=\"color: #0550ae\">2</span><span style=\"color: #24292f;background-color: #f6f8fa\">];</span>\n</div><div class='code-line code-line-20'>\n</div><div class='code-line code-line-21'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">input_text</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #cf222e\">match</span>&nbsp;<span style=\"color: #953800\">fs</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #8250df\">read_to_string</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">input_file</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-22'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">Ok</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">content</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #cf222e\">=&gt;</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">content</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>\n</div><div class='code-line code-line-23'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">Err</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">e</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #cf222e\">=&gt;</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-24'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">eprintln!</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">\"Error reading input file: {}\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">e</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-25'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">std</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #953800\">process</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #8250df\">exit</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">1</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-26'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-27'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">};</span>\n</div><div class='code-line code-line-28'>\n</div><div class='code-line code-line-29'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">model</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #953800\">TextEmbedding</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #8250df\">try_new</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span>\n</div><div class='code-line code-line-30'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">InitOptions</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #8250df\">new</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #953800\">EmbeddingModel</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #24292f;background-color: #f6f8fa\">AllMiniLML6V2</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span><span style=\"color: #8250df\">.with_show_download_progress</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #cf222e\">true</span><span style=\"color: #24292f;background-color: #f6f8fa\">),</span>\n</div><div class='code-line code-line-31'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">)</span><span style=\"color: #8250df\">.expect</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">\"Failed to initialize the embedding model\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-32'>\n</div><div class='code-line code-line-33'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">documents</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #8250df\">vec!</span><span style=\"color: #24292f;background-color: #f6f8fa\">[</span><span style=\"color: #24292f;background-color: #f6f8fa\">input_text</span><span style=\"color: #24292f;background-color: #f6f8fa\">];</span>\n</div><div class='code-line code-line-34'>\n</div><div class='code-line code-line-35'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embeddings</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #cf222e\">match</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">model</span><span style=\"color: #8250df\">.embed</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">documents</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #953800\">None</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-36'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">Ok</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">embeddings</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #cf222e\">=&gt;</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embeddings</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>\n</div><div class='code-line code-line-37'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">Err</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">e</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #cf222e\">=&gt;</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-38'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">eprintln!</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">\"Error generating embedding: {}\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">e</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-39'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">std</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #953800\">process</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #8250df\">exit</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">1</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-40'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-41'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">};</span>\n</div><div class='code-line code-line-42'>\n</div><div class='code-line code-line-43'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">_embedding</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embeddings</span><span style=\"color: #24292f;background-color: #f6f8fa\">[</span><span style=\"color: #0550ae\">0</span><span style=\"color: #24292f;background-color: #f6f8fa\">][</span><span style=\"color: #0550ae\">0</span><span style=\"color: #24292f;background-color: #f6f8fa\">];</span>\n</div><div class='code-line code-line-44'>\n</div><div class='code-line code-line-45'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">_embedding_str</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embeddings</span><span style=\"color: #24292f;background-color: #f6f8fa\">[</span><span style=\"color: #0550ae\">0</span><span style=\"color: #24292f;background-color: #f6f8fa\">]</span>\n</div><div class='code-line code-line-46'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">.iter</span><span style=\"color: #24292f;background-color: #f6f8fa\">()</span>\n</div><div class='code-line code-line-47'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">.map</span><span style=\"color: #24292f;background-color: #f6f8fa\">(|</span><span style=\"color: #24292f;background-color: #f6f8fa\">f</span><span style=\"color: #24292f;background-color: #f6f8fa\">|</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">f</span><span style=\"color: #8250df\">.to_string</span><span style=\"color: #24292f;background-color: #f6f8fa\">())</span>\n</div><div class='code-line code-line-48'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #0550ae\">.collect</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #0550ae\">&lt;</span><span style=\"color: #953800\">Vec</span><span style=\"color: #0550ae\">&lt;</span><span style=\"color: #24292f;background-color: #f6f8fa\">_</span><span style=\"color: #0550ae\">&gt;&gt;</span><span style=\"color: #24292f;background-color: #f6f8fa\">()</span>\n</div><div class='code-line code-line-49'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">.join</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">\",\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-50'>\n</div><div class='code-line code-line-51'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embedding_str</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #8250df\">format!</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">\"{}{}{}\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #0a3069\">\"[\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">_embedding_str</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #0a3069\">\"]\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-52'>\n</div><div class='code-line code-line-53'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">json</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #953800\">serde_json</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #8250df\">to_string</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">&amp;</span><span style=\"color: #24292f;background-color: #f6f8fa\">EmbeddingOutput</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-54'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embedding</span><span style=\"color: #24292f;background-color: #f6f8fa\">:</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">embedding_str</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>\n</div><div class='code-line code-line-55'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">})</span><span style=\"color: #8250df\">.unwrap</span><span style=\"color: #24292f;background-color: #f6f8fa\">();</span>\n</div><div class='code-line code-line-56'>\n</div><div class='code-line code-line-57'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//match fs::write(output_file, json) {</span>\n</div><div class='code-line code-line-58'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//Ok(_) | Err(_) =&gt; todo!(),</span>\n</div><div class='code-line code-line-59'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//Ok(_) =&gt; eprintln!(\"Embedding written to {}\", output_file),</span>\n</div><div class='code-line code-line-60'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//Err(e) =&gt; eprintln!(\"Error writing to output file: {}\", e),</span>\n</div><div class='code-line code-line-61'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #6e7781\">//}</span>\n</div><div class='code-line code-line-62'>\n</div><div class='code-line code-line-63'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #cf222e\">if</span>&nbsp;<span style=\"color: #cf222e\">let</span>&nbsp;<span style=\"color: #8250df\">Err</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">e</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #0550ae\">=</span>&nbsp;<span style=\"color: #953800\">fs</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #8250df\">write</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #24292f;background-color: #f6f8fa\">output_file</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">json</span><span style=\"color: #24292f;background-color: #f6f8fa\">)</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">{</span>\n</div><div class='code-line code-line-64'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #8250df\">eprintln!</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0a3069\">\"Error writing to output file: {}\"</span><span style=\"color: #24292f;background-color: #f6f8fa\">,</span>&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">e</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-65'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #953800\">std</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #953800\">process</span><span style=\"color: #24292f;background-color: #f6f8fa\">::</span><span style=\"color: #8250df\">exit</span><span style=\"color: #24292f;background-color: #f6f8fa\">(</span><span style=\"color: #0550ae\">1</span><span style=\"color: #24292f;background-color: #f6f8fa\">);</span>\n</div><div class='code-line code-line-66'>&nbsp;&nbsp;&nbsp;&nbsp;<span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-67'><span style=\"color: #24292f;background-color: #f6f8fa\">}</span>\n</div><div class='code-line code-line-68'>\n</div></code>\n</div>\n"
+        }
+      },
+      "_meta": {
+        "identifier": "nerd/fastest-possible-embeddings",
+        "type": "nerd",
+        "id": "fastest-possible-embeddings",
+        "urls": [
+          "/ro/nerd/fastest-possible-embeddings/assets/giraffe.jpeg",
+          "/ro/nerd/fastest-possible-embeddings/assets/let-me-in.jpg",
+          "/ro/nerd/fastest-possible-embeddings/assets/src/fastembed.js",
+          "/ro/nerd/fastest-possible-embeddings/assets/src/fastembed.rs",
+          "/ro/nerd/fastest-possible-embeddings/assets/terminal.jpg",
+          "/ro/nerd/fastest-possible-embeddings/attributes.yml",
+          "/ro/nerd/fastest-possible-embeddings/body.md"
+        ],
+        "created_at": "2025-04-23 04:38:29 +0000",
+        "updated_at": "2025-04-23 04:38:29 +0000",
+        "rel": {
+          "curr": "nerd/fastest-possible-embeddings",
+          "prev": null,
+          "next": "nerd/ima"
+        }
+      }
+    }
+  },
+  "_meta": {
+    "url": "/ro",
+    "type": "nerd",
+    "id": "fastest-possible-embeddings"
+  }
+}

data/public/api/ro/nerd/ima/index.json

@@ -0,0 +1,49 @@
+{
+  "data": {
+    "nerd/ima": {
+      "og": {
+        "title": "ima /code/fixme && /write/it",
+        "description": "`ima` is a universal unix filter, prompt library, coding helper, and wrtiting assistant that uses mistral and groq AI for hella fast, cheap, and simple day-to-day tasks",
+        "image": {
+          "url": "/ro/nerd/ima/assets/og.jpeg"
+        }
+      },
+      "body": "<div class=\"ro markdown\">\n  <p><code>ima</code> is the universal unix filter.</p>\n\n<p>it is one part code generator, one part writing assistant, one part magic cli.</p>\n\n<p>and one part super uber ultra tiny:  currently weighing in at about 250 lines of code\nwhile offering 80% of the functionality of insanely large, expensive, and\nprivacy violating ide plugins.</p>\n\n<p>it is under active development and requires no dependencies except ruby and a\ngroq api key.  if you don’t understand the difference between ‘inference’ and\n‘training’ best move along… nothing to see.  keep shipping your .env files\nto microsoft and openai.  no really ;-)</p>\n\n<p>super short sneak peaks for now:</p>\n\n<ol>\n  <li><a href=\"https://photos.app.goo.gl/lgbtthvrywtubvgr9\">code and prose editing with shortcodes</a></li>\n  <li><a href=\"https://photos.app.goo.gl/d3hopktecraq4cxls8\">the do <em>anything</em> unix filter mode</a></li>\n  <li><a href=\"https://photos.app.goo.gl/ldd1tyaj5zsde1qo59\">using local context</a></li>\n</ol>\n\n<p>you may have noticed but.. to date, <strong>259 lines of code</strong></p>\n\n<p>it will get bigger but, not by much <a href=\"/contact\">reach out</a> to learn more.</p>\n\n</div>\n",
+      "assets": {
+        "assets/og.jpeg": {
+          "url": "/ro/nerd/ima/assets/og.jpeg",
+          "path": "nerd/ima/assets/og.jpeg",
+          "size": 8,
+          "img": {
+            "format": "jpeg",
+            "width": 2048,
+            "height": 2048
+          },
+          "src": null
+        }
+      },
+      "_meta": {
+        "identifier": "nerd/ima",
+        "type": "nerd",
+        "id": "ima",
+        "urls": [
+          "/ro/nerd/ima/assets/og.jpeg",
+          "/ro/nerd/ima/attributes.yml",
+          "/ro/nerd/ima/body.md"
+        ],
+        "created_at": "2025-04-23 04:38:29 +0000",
+        "updated_at": "2025-04-23 04:38:29 +0000",
+        "rel": {
+          "curr": "nerd/ima",
+          "prev": "nerd/fastest-possible-embeddings",
+          "next": "nerd/index"
+        }
+      }
+    }
+  },
+  "_meta": {
+    "url": "/ro",
+    "type": "nerd",
+    "id": "ima"
+  }
+}

data/public/api/ro/nerd/index/index.json

@@ -0,0 +1,74 @@
+{
+  "data": {
+    "nerd/index": {
+      "og": {
+        "title": "Big fat 🤓",
+        "description": "I have grown so tall!",
+        "image": {
+          "url": "/ro/nerd/index/assets/giraffe.jpeg",
+          "width": 2048
+        }
+      },
+      "body": "<div class=\"ro markdown\">\n  <h3 id=\"they-say-i-am-a-big-fat---and-they-are-right\">they say i am a big fat 🤓 – and they are right!</h3>\n\n<p>for some strange reason, this makes me happy:</p>\n\n<p><img src='/ro/nerd/index/assets/terminal.jpg' alt=\"\" title=\"image of terminal\" /></p>\n\n<p>as you can read on my <a href=\"/about\">about</a> page, i have written <a href=\"/rubygems\">way too much software</a>.</p>\n\n<p>i got my start researching <em>in</em> <a href=\"https://www.boulder.noaa.gov/\">NOAA</a></p>\n\n<p><em>for</em> <a href=\"https://cires.colorado.edu/\">C.I.R.E.S</a></p>\n\n<p>while studying <em>at</em> <a href=\"https://www.colorado.edu/engineering/\">CU’s College of Engineering &amp; Applied Science</a></p>\n\n<blockquote>\n  <p>wut?</p>\n</blockquote>\n\n<p>basically, the university has a program that donates young scientists to other\nresearch institutes, to help do science.</p>\n\n<p>its goal is getting the university’s name on papers which, if you know\nanything about science, is gold.  publish or perish… etc.  publishing ==\nfunding.</p>\n\n<p><strong>30</strong> years later, i realize that this was a fantastic introduction to start-up\nculture.  no b.s., just, make shit that works, and <em>go</em>.  no one to tell you\nwhat <em>‘not to do’</em> or market signals that design your product for you so you\ndon’t have to actually think and be bold - just raw <em>instinct</em> about what\nshould be studied, how, and <em>why</em>.</p>\n\n<p>that, and fundraising… nothing like buildig stuff and figuring out how to\npay for it at the same time  ;-)</p>\n\n<p>the first project i did at CIRES, is still one of my favorite projects of all\ntime: we wrote a system, designed to run on old-skool linux field computers,\nthat forest fire fighters would use, tactically, in the field, to decide\nweather/whether or not (pun intended), sending a crew up a canyon to battle\nthe blaze would result in them dying.  mainly it was a wind analysis tool,\nhyper local weather, delivered to a device, long before iphones became a\nthing.</p>\n\n<p>(this is my explanation for why, when the <a href=\"https://photos.app.goo.gl/Kjipk8eKT88CtSRSA\">los angeles fires</a>\n erupted i hopped right on my bike and went to check them out… fires and the jobs responders\nare required to do, for $26/hr, astounds me)</p>\n\n<p>subsequently, i went to work at FSL (Forecast Systems Lab) doing\nhyper-high-availability (5 9s ((99.999 % uptime))) for operational satellite\ningest systems.</p>\n\n<p>we designed cutting edge systems. and novel… brutal… methods of ensuring\nconsistency of classified data such as STONITH, which stands for “Stone The\nOther In The Head”, a method used in what were then cutting edge\nhigh-availibilty clusters that would manage taking over as ‘master’ (a term\nsince banished from software, probably for the best…) by literally toggling\nthe power of the other node, to be damn sure it was off.  things we simpler\nthen, but also very complex.  there was a lot to invent on every project.\nsass wasn’t even a word.</p>\n\n<p>i also did a lot of work in model verification: geophysical models take\nhundreds, or thousands, or even hundreds of thousands, of configurations to\nrun.  people talk about how neat 12-factor configuration is now, and i just\nshake my head… what if you had to manage millions of configuration values?\nthe next trick is version them, so we know how they change over time because,\nas scientists, if we make a change to say, a cloud physics model, we need to\n‘test it’.  but</p>\n\n<blockquote>\n  <p>how <em>do</em> you test software, when you don’t now the ‘right answer’?</p>\n</blockquote>\n\n<p>the approach is actually, theoretically, simple:</p>\n\n<p>you hold all variables, all the hundreds of thousands of them, constant, make\nchanges to a few, and then look for patterns of changes in the output.  in the\ncase of weather models, this could be mean that a change to a cloud physics\nmodel resulted in predicting 8/9 historical storms with accuracy, vs. the 7/9\na previous iteration would have predicted.</p>\n\n<p>this type of analysis, foreign to many engineers, is back with vengance,</p>\n\n<p>thanks to <a href=\"/disco\">AI</a>…</p>\n\n<p>my next stint was at <a href=\"https://www.ngdc.noaa.gov/\">The National Geophysical Data Center</a>, where i was able to participate in a bunch of super cool research:</p>\n\n<ul>\n  <li><a href=\"/purls/can-poverty-rates-be-estimated-using-satellite-data\">/purls/can-poverty-rates-be-estimated-using-satellite-data</a></li>\n  <li><a href=\"/purls/a-global-inventory-of-coral-reef-stressors-based-on-satellite-observed-nighttime-lights\">/purls/a-global-inventory-of-coral-reef-stressors-based-on-satellite-observed-nighttime-lights</a></li>\n  <li><a href=\"/purls/change-detection-in-satellite-observed-nighttime-lights-1992-2003\">/purls/change-detection-in-satellite-observed-nighttime-lights-1992-2003</a></li>\n  <li><a href=\"/purls/a-twelve-year-record-of-national-and-global-gas-flaring-volumes-estimated-using-satellite-data\">/purls/a-twelve-year-record-of-national-and-global-gas-flaring-volumes-estimated-using-satellite-data</a></li>\n  <li><a href=\"/purls/global-distribution-and-density-of-constructed-impervious-surfaces\">/purls/global-distribution-and-density-of-constructed-impervious-surfaces</a></li>\n</ul>\n\n<p>and built very, very <a href=\"https://www.linuxjournal.com/article/7922\">large super-compute</a>,\nessentially big fat map-reduce style computing but, at the time, neither of\nthose terms existed.  we had to invent novel ways, of moving our code of off\nbig-endian (not spelled wrong) cray (also not spelled wrong) machines and onto\ntons of commodity hardware.  namely, hundreds of linux boxen.</p>\n\n<p>i also did a ton of work around clustering… very low level c/c++ code, using\nideas from signal processing and computer vision, to detect the edges of cities\nvia a process similar to the <a href=\"https://en.wikipedia.org/wiki/Watershed_(image_processing)\">watershed algorithm</a>\n but…  at scale.</p>\n\n<p>throughout my tenure at NGDC, i was allowed to release piles of open source\nsoftware and, i am very, very grateful for this.  eventually i was able to\nshare, through oss, over 200 open source projects enjoyed by many.  i think\nthis was foundational to my eventually winning a ‘ruby hero’ award and wish\nthat more young engineers had creative time to just build things.   this, is\nwhere true innovation comes from i believe.  not board rooms or from mining\nthe data to just give people what they want.  which, is probably potato chips.</p>\n\n<hr />\n\n<p><strong>coffee break…</strong></p>\n<hr />\n\n<p>next, <a href=\"https://www.linkedin.com/in/gregory-greenstreet-082635/\">this cowboy</a>\nhired me: to compile the <a href=\"https://www.gnu.org/software/gsl/\">GNU scientific library</a> on.. wait for it…  <strong>windows</strong>!</p>\n\n<p>yep, i am <em>that</em> old!</p>\n\n<p>(strange that, for the first time ever, i would actually now consider owning a\nmicro$oft computer but… only because they run linux ;-)</p>\n\n<p>anyhow, Greg worked for <a href=\"https://www.linkedin.com/in/dospringer/\">Don Springer</a>, at company called <a href=\"https://www.oracle.com/corporate/pressrelease/oracle-buys-collective-intellect-060512.html\">Collective Intellect</a>.  which, at the time, was the “Mobius Group” (which would eventually become <a href=\"https://foundry.vc/\">The Foundry Group</a> and… <strong>#BOOM</strong> .. start-ups in Boulder, Colorado, were a thing.</p>\n\n<p>it was fun time.</p>\n\n<p>it was after this that i started <a href=\"/dojo4\">dojo4</a>, which was the crown jewl in my life as a geek, for many reasons i hope to write about soon.  including close to ten years mentoring <a href=\"https://www.techstars.com/\">techstars</a> companies where, i have made some super duper great friends.</p>\n\n<p>until then, i will say, as i always do that:</p>\n\n<ul>\n  <li>this is all, always, a work in progress.</li>\n  <li>i am doing it live.</li>\n  <li>i cannot spell, so sue me.  (a little dislexic, in fact.)</li>\n</ul>\n\n</div>\n",
+      "assets": {
+        "assets/giraffe.jpeg": {
+          "url": "/ro/nerd/index/assets/giraffe.jpeg",
+          "path": "nerd/index/assets/giraffe.jpeg",
+          "size": 8,
+          "img": {
+            "format": "jpeg",
+            "width": 2048,
+            "height": 2048
+          },
+          "src": null
+        },
+        "assets/let-me-in.jpg": {
+          "url": "/ro/nerd/index/assets/let-me-in.jpg",
+          "path": "nerd/index/assets/let-me-in.jpg",
+          "size": 8,
+          "img": {
+            "format": "jpeg",
+            "width": 1920,
+            "height": 1200
+          },
+          "src": null
+        },
+        "assets/terminal.jpg": {
+          "url": "/ro/nerd/index/assets/terminal.jpg",
+          "path": "nerd/index/assets/terminal.jpg",
+          "size": 8,
+          "img": {
+            "format": "jpeg",
+            "width": 1201,
+            "height": 1194
+          },
+          "src": null
+        }
+      },
+      "_meta": {
+        "identifier": "nerd/index",
+        "type": "nerd",
+        "id": "index",
+        "urls": [
+          "/ro/nerd/index/assets/giraffe.jpeg",
+          "/ro/nerd/index/assets/let-me-in.jpg",
+          "/ro/nerd/index/assets/terminal.jpg",
+          "/ro/nerd/index/attributes.yml",
+          "/ro/nerd/index/body.md"
+        ],
+        "created_at": "2025-04-23 04:38:29 +0000",
+        "updated_at": "2025-04-23 04:38:29 +0000",
+        "rel": {
+          "curr": "nerd/index",
+          "prev": "nerd/ima",
+          "next": null
+        }
+      }
+    }
+  },
+  "_meta": {
+    "url": "/ro",
+    "type": "nerd",
+    "id": "index"
+  }
+}