lex-cognitive-compression 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.github/workflows/ci.yml +16 -0
- data/.gitignore +2 -0
- data/.rspec +4 -0
- data/.rubocop.yml +40 -0
- data/CLAUDE.md +73 -0
- data/Gemfile +13 -0
- data/Gemfile.lock +78 -0
- data/README.md +45 -0
- data/lex-cognitive-compression.gemspec +31 -0
- data/lib/legion/extensions/cognitive_compression/client.rb +11 -0
- data/lib/legion/extensions/cognitive_compression/helpers/compression_engine.rb +141 -0
- data/lib/legion/extensions/cognitive_compression/helpers/constants.rb +40 -0
- data/lib/legion/extensions/cognitive_compression/helpers/information_chunk.rb +77 -0
- data/lib/legion/extensions/cognitive_compression/runners/cognitive_compression.rb +72 -0
- data/lib/legion/extensions/cognitive_compression/version.rb +9 -0
- data/lib/legion/extensions/cognitive_compression.rb +15 -0
- metadata +77 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 2267287b5f2dea75cb6cb8606b891280b9a8593f1a83febdd769747cc2254ebf
|
|
4
|
+
data.tar.gz: 90fd47f51582ff7ad354b7a3885958e078dbd73db3696e6b96d731a7a01fbecd
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: c55c3ec947051588b37b93c7d5ea4de4c4070d5cd9c9377980c8aa62ba111b57b7528c6cd492a34c4463f94e68730741d8673af36c662b0c3d7777a6350c48ee
|
|
7
|
+
data.tar.gz: 8abc5bea22c7538d9ccadfad5f94a2efb4591714d3a45e71b52e50c387b35664b3f46c59ce7966b851f89e5bb9ac01f44471778e67197fbca445ca9b14dc5c2d
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
on:
|
|
3
|
+
push:
|
|
4
|
+
branches: [main]
|
|
5
|
+
pull_request:
|
|
6
|
+
|
|
7
|
+
jobs:
|
|
8
|
+
ci:
|
|
9
|
+
uses: LegionIO/.github/.github/workflows/ci.yml@main
|
|
10
|
+
|
|
11
|
+
release:
|
|
12
|
+
needs: ci
|
|
13
|
+
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
|
|
14
|
+
uses: LegionIO/.github/.github/workflows/release.yml@main
|
|
15
|
+
secrets:
|
|
16
|
+
rubygems-api-key: ${{ secrets.RUBYGEMS_API_KEY }}
|
data/.gitignore
ADDED
data/.rspec
ADDED
data/.rubocop.yml
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
AllCops:
|
|
2
|
+
NewCops: enable
|
|
3
|
+
TargetRubyVersion: 3.4
|
|
4
|
+
|
|
5
|
+
Style/Documentation:
|
|
6
|
+
Enabled: false
|
|
7
|
+
|
|
8
|
+
Naming/PredicateMethod:
|
|
9
|
+
Enabled: false
|
|
10
|
+
|
|
11
|
+
Metrics/ClassLength:
|
|
12
|
+
Max: 150
|
|
13
|
+
|
|
14
|
+
Metrics/MethodLength:
|
|
15
|
+
Max: 25
|
|
16
|
+
|
|
17
|
+
Metrics/AbcSize:
|
|
18
|
+
Max: 25
|
|
19
|
+
|
|
20
|
+
Metrics/ParameterLists:
|
|
21
|
+
Max: 8
|
|
22
|
+
MaxOptionalParameters: 8
|
|
23
|
+
|
|
24
|
+
Layout/HashAlignment:
|
|
25
|
+
EnforcedColonStyle: table
|
|
26
|
+
EnforcedHashRocketStyle: table
|
|
27
|
+
|
|
28
|
+
Metrics/BlockLength:
|
|
29
|
+
Exclude:
|
|
30
|
+
- 'spec/**/*'
|
|
31
|
+
|
|
32
|
+
Style/FrozenStringLiteralComment:
|
|
33
|
+
Enabled: true
|
|
34
|
+
|
|
35
|
+
Style/StringLiterals:
|
|
36
|
+
EnforcedStyle: single_quotes
|
|
37
|
+
|
|
38
|
+
Style/OneClassPerFile:
|
|
39
|
+
Exclude:
|
|
40
|
+
- 'spec/spec_helper.rb'
|
data/CLAUDE.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
# lex-cognitive-compression
|
|
2
|
+
|
|
3
|
+
**Level 3 Documentation**
|
|
4
|
+
- **Parent**: `/Users/miverso2/rubymine/legion/extensions-agentic/CLAUDE.md`
|
|
5
|
+
- **Grandparent**: `/Users/miverso2/rubymine/legion/CLAUDE.md`
|
|
6
|
+
|
|
7
|
+
## Purpose
|
|
8
|
+
|
|
9
|
+
Models information compression and abstraction as knowledge moves through memory tiers. Raw episodic content is compressed as it ages; fidelity decreases with compression, but storage cost drops. Abstraction groups chunks into higher-level representations. Mirrors the psychological phenomenon where detailed memories fade to gist-level representations over time.
|
|
10
|
+
|
|
11
|
+
## Gem Info
|
|
12
|
+
|
|
13
|
+
- **Gem name**: `lex-cognitive-compression`
|
|
14
|
+
- **Version**: `0.1.0`
|
|
15
|
+
- **Module**: `Legion::Extensions::CognitiveCompression`
|
|
16
|
+
- **Ruby**: `>= 3.4`
|
|
17
|
+
- **License**: MIT
|
|
18
|
+
|
|
19
|
+
## File Structure
|
|
20
|
+
|
|
21
|
+
```
|
|
22
|
+
lib/legion/extensions/cognitive_compression/
|
|
23
|
+
cognitive_compression.rb
|
|
24
|
+
version.rb
|
|
25
|
+
client.rb
|
|
26
|
+
helpers/
|
|
27
|
+
constants.rb
|
|
28
|
+
compression_engine.rb
|
|
29
|
+
information_chunk.rb
|
|
30
|
+
runners/
|
|
31
|
+
cognitive_compression.rb
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Key Constants
|
|
35
|
+
|
|
36
|
+
From `helpers/constants.rb`:
|
|
37
|
+
|
|
38
|
+
- `CHUNK_TYPES` — `%i[episodic semantic procedural sensory emotional abstract relational]`
|
|
39
|
+
- `MAX_CHUNKS` = `500`, `MAX_ABSTRACTIONS` = `200`
|
|
40
|
+
- `DEFAULT_COMPRESSION_RATIO` = `0.5`, `COMPRESSION_RATE` = `0.1`
|
|
41
|
+
- `FIDELITY_LOSS_RATE` = `0.02`, `MIN_FIDELITY` = `0.1`
|
|
42
|
+
- `COMPRESSION_LABELS` — `0.8+` = `:highly_compressed`, `0.6` = `:compressed`, `0.4` = `:moderate`, `0.2` = `:detailed`, below = `:raw`
|
|
43
|
+
- `FIDELITY_LABELS` — `0.8+` = `:pristine`, `0.6` = `:faithful`, `0.4` = `:approximate`, `0.2` = `:lossy`, below = `:degraded`
|
|
44
|
+
|
|
45
|
+
## Runners
|
|
46
|
+
|
|
47
|
+
All methods in `Runners::CognitiveCompression`:
|
|
48
|
+
|
|
49
|
+
- `store_chunk(label:, chunk_type: :semantic, original_size: 1.0)` — stores a new chunk at default compression ratio
|
|
50
|
+
- `compress_chunk(chunk_id:, amount: COMPRESSION_RATE)` — increases compression, decreasing fidelity
|
|
51
|
+
- `decompress_chunk(chunk_id:, amount: COMPRESSION_RATE)` — decreases compression, partially restoring fidelity
|
|
52
|
+
- `abstract_chunks(chunk_ids:, abstraction_label:)` — combines multiple chunks into a single higher-level abstraction
|
|
53
|
+
- `compress_all(amount: COMPRESSION_RATE)` — applies compression to all chunks; intended as periodic runner
|
|
54
|
+
- `average_fidelity` — mean fidelity across all stored chunks
|
|
55
|
+
- `overall_compression_ratio` — mean compression ratio
|
|
56
|
+
- `compression_report` — full report: chunk counts, fidelity, compression distribution, abstraction count
|
|
57
|
+
|
|
58
|
+
## Helpers
|
|
59
|
+
|
|
60
|
+
- `CompressionEngine` — manages `@chunks` and `@abstractions`. `abstract_chunks` creates a new `InformationChunk` representing the merged abstraction, linking back to source chunk IDs.
|
|
61
|
+
- `InformationChunk` — has `chunk_type`, `original_size`, `compression_ratio`, `fidelity`. `compress!(amount)` increases ratio and decreases fidelity. `decompress!(amount)` reverses compression but fidelity does not fully recover (capped at original minus `FIDELITY_LOSS_RATE * compressions`).
|
|
62
|
+
|
|
63
|
+
## Integration Points
|
|
64
|
+
|
|
65
|
+
- `lex-memory` handles long-term trace storage with tier migration; compression models what happens to fidelity as traces move from `episodic` to `semantic` tier — detail is lost, gist is retained.
|
|
66
|
+
- `lex-cognitive-chunking` creates named groups; compression models what happens to those groups over time as they age.
|
|
67
|
+
- `compress_all` is the natural periodic maintenance runner — called each decay cycle alongside `lex-memory`'s `decay_cycle`.
|
|
68
|
+
|
|
69
|
+
## Development Notes
|
|
70
|
+
|
|
71
|
+
- Decompression is lossy: `fidelity` does not return to 1.0 after repeated compress/decompress cycles. `MIN_FIDELITY = 0.1` is a floor; once reached, no further fidelity loss occurs.
|
|
72
|
+
- `abstract_chunks` returns `nil` if no valid source chunks are found — the runner wraps this with `{ success: false, error: 'no valid chunks' }`.
|
|
73
|
+
- `chunk_type: :episodic` represents autobiographical/event memory (high initial fidelity, fast compression target). `chunk_type: :semantic` represents general knowledge (moderate compression acceptable).
|
data/Gemfile
ADDED
data/Gemfile.lock
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
PATH
|
|
2
|
+
remote: .
|
|
3
|
+
specs:
|
|
4
|
+
lex-cognitive-compression (0.1.0)
|
|
5
|
+
|
|
6
|
+
GEM
|
|
7
|
+
remote: https://rubygems.org/
|
|
8
|
+
specs:
|
|
9
|
+
addressable (2.8.9)
|
|
10
|
+
public_suffix (>= 2.0.2, < 8.0)
|
|
11
|
+
ast (2.4.3)
|
|
12
|
+
bigdecimal (4.0.1)
|
|
13
|
+
diff-lcs (1.6.2)
|
|
14
|
+
json (2.19.1)
|
|
15
|
+
json-schema (6.2.0)
|
|
16
|
+
addressable (~> 2.8)
|
|
17
|
+
bigdecimal (>= 3.1, < 5)
|
|
18
|
+
language_server-protocol (3.17.0.5)
|
|
19
|
+
lint_roller (1.1.0)
|
|
20
|
+
mcp (0.8.0)
|
|
21
|
+
json-schema (>= 4.1)
|
|
22
|
+
parallel (1.27.0)
|
|
23
|
+
parser (3.3.10.2)
|
|
24
|
+
ast (~> 2.4.1)
|
|
25
|
+
racc
|
|
26
|
+
prism (1.9.0)
|
|
27
|
+
public_suffix (7.0.5)
|
|
28
|
+
racc (1.8.1)
|
|
29
|
+
rainbow (3.1.1)
|
|
30
|
+
regexp_parser (2.11.3)
|
|
31
|
+
rspec (3.13.2)
|
|
32
|
+
rspec-core (~> 3.13.0)
|
|
33
|
+
rspec-expectations (~> 3.13.0)
|
|
34
|
+
rspec-mocks (~> 3.13.0)
|
|
35
|
+
rspec-core (3.13.6)
|
|
36
|
+
rspec-support (~> 3.13.0)
|
|
37
|
+
rspec-expectations (3.13.5)
|
|
38
|
+
diff-lcs (>= 1.2.0, < 2.0)
|
|
39
|
+
rspec-support (~> 3.13.0)
|
|
40
|
+
rspec-mocks (3.13.8)
|
|
41
|
+
diff-lcs (>= 1.2.0, < 2.0)
|
|
42
|
+
rspec-support (~> 3.13.0)
|
|
43
|
+
rspec-support (3.13.7)
|
|
44
|
+
rubocop (1.85.1)
|
|
45
|
+
json (~> 2.3)
|
|
46
|
+
language_server-protocol (~> 3.17.0.2)
|
|
47
|
+
lint_roller (~> 1.1.0)
|
|
48
|
+
mcp (~> 0.6)
|
|
49
|
+
parallel (~> 1.10)
|
|
50
|
+
parser (>= 3.3.0.2)
|
|
51
|
+
rainbow (>= 2.2.2, < 4.0)
|
|
52
|
+
regexp_parser (>= 2.9.3, < 3.0)
|
|
53
|
+
rubocop-ast (>= 1.49.0, < 2.0)
|
|
54
|
+
ruby-progressbar (~> 1.7)
|
|
55
|
+
unicode-display_width (>= 2.4.0, < 4.0)
|
|
56
|
+
rubocop-ast (1.49.1)
|
|
57
|
+
parser (>= 3.3.7.2)
|
|
58
|
+
prism (~> 1.7)
|
|
59
|
+
rubocop-rspec (3.9.0)
|
|
60
|
+
lint_roller (~> 1.1)
|
|
61
|
+
rubocop (~> 1.81)
|
|
62
|
+
ruby-progressbar (1.13.0)
|
|
63
|
+
unicode-display_width (3.2.0)
|
|
64
|
+
unicode-emoji (~> 4.1)
|
|
65
|
+
unicode-emoji (4.2.0)
|
|
66
|
+
|
|
67
|
+
PLATFORMS
|
|
68
|
+
arm64-darwin-25
|
|
69
|
+
ruby
|
|
70
|
+
|
|
71
|
+
DEPENDENCIES
|
|
72
|
+
lex-cognitive-compression!
|
|
73
|
+
rspec (~> 3.13)
|
|
74
|
+
rubocop (~> 1.75)
|
|
75
|
+
rubocop-rspec
|
|
76
|
+
|
|
77
|
+
BUNDLED WITH
|
|
78
|
+
2.6.9
|
data/README.md
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# lex-cognitive-compression
|
|
2
|
+
|
|
3
|
+
Cognitive information compression for LegionIO. Models information compression and abstraction as knowledge moves through memory tiers.
|
|
4
|
+
|
|
5
|
+
## What It Does
|
|
6
|
+
|
|
7
|
+
Raw memories start with high fidelity but take full storage. Over time they compress — detail fades, gist remains. This extension models that process: chunks are stored at a compression ratio, can be incrementally compressed (losing fidelity), and can be abstracted into higher-level representations. Decompression is lossy — you cannot fully recover the original fidelity once lost.
|
|
8
|
+
|
|
9
|
+
Seven chunk types are supported: episodic, semantic, procedural, sensory, emotional, abstract, and relational. Each has different natural compression targets appropriate to its type.
|
|
10
|
+
|
|
11
|
+
## Usage
|
|
12
|
+
|
|
13
|
+
```ruby
|
|
14
|
+
client = Legion::Extensions::CognitiveCompression::Client.new
|
|
15
|
+
|
|
16
|
+
chunk = client.store_chunk(
|
|
17
|
+
label: 'meeting notes 2026-03-14',
|
|
18
|
+
chunk_type: :episodic,
|
|
19
|
+
original_size: 1.0
|
|
20
|
+
)
|
|
21
|
+
|
|
22
|
+
client.compress_chunk(chunk_id: chunk[:chunk][:id], amount: 0.2)
|
|
23
|
+
client.average_fidelity
|
|
24
|
+
# => { fidelity: 0.96 } # small fidelity loss from one compression step
|
|
25
|
+
|
|
26
|
+
# Abstract multiple chunks into a summary
|
|
27
|
+
client.abstract_chunks(
|
|
28
|
+
chunk_ids: [chunk[:chunk][:id]],
|
|
29
|
+
abstraction_label: 'Q1 meeting themes'
|
|
30
|
+
)
|
|
31
|
+
|
|
32
|
+
client.compression_report
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
## Development
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
bundle install
|
|
39
|
+
bundle exec rspec
|
|
40
|
+
bundle exec rubocop
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## License
|
|
44
|
+
|
|
45
|
+
MIT
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative 'lib/legion/extensions/cognitive_compression/version'
|
|
4
|
+
|
|
5
|
+
Gem::Specification.new do |spec|
|
|
6
|
+
spec.name = 'lex-cognitive-compression'
|
|
7
|
+
spec.version = Legion::Extensions::CognitiveCompression::VERSION
|
|
8
|
+
spec.authors = ['Esity']
|
|
9
|
+
spec.email = ['matthewdiverson@gmail.com']
|
|
10
|
+
|
|
11
|
+
spec.summary = 'Cognitive information compression for LegionIO'
|
|
12
|
+
spec.description = 'Models information compression and abstraction as knowledge moves through memory tiers'
|
|
13
|
+
spec.homepage = 'https://github.com/LegionIO/lex-cognitive-compression'
|
|
14
|
+
spec.license = 'MIT'
|
|
15
|
+
spec.required_ruby_version = '>= 3.4'
|
|
16
|
+
|
|
17
|
+
spec.metadata['homepage_uri'] = spec.homepage
|
|
18
|
+
spec.metadata['source_code_uri'] = spec.homepage
|
|
19
|
+
spec.metadata['documentation_uri'] = "#{spec.homepage}/blob/main/README.md"
|
|
20
|
+
spec.metadata['changelog_uri'] = "#{spec.homepage}/blob/main/CHANGELOG.md"
|
|
21
|
+
spec.metadata['bug_tracker_uri'] = "#{spec.homepage}/issues"
|
|
22
|
+
spec.metadata['rubygems_mfa_required'] = 'true'
|
|
23
|
+
|
|
24
|
+
spec.files = Dir.chdir(__dir__) do
|
|
25
|
+
`git ls-files -z`.split("\x0").reject do |f|
|
|
26
|
+
f.match(%r{\A(?:test|spec|features)/})
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
spec.require_paths = ['lib']
|
|
30
|
+
spec.add_development_dependency 'legion-gaia'
|
|
31
|
+
end
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Legion
|
|
4
|
+
module Extensions
|
|
5
|
+
module CognitiveCompression
|
|
6
|
+
module Helpers
|
|
7
|
+
class CompressionEngine
|
|
8
|
+
include Constants
|
|
9
|
+
|
|
10
|
+
def initialize
|
|
11
|
+
@chunks = {}
|
|
12
|
+
@abstractions = {}
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def store_chunk(label:, chunk_type: :semantic, original_size: 1.0)
|
|
16
|
+
prune_chunks_if_needed
|
|
17
|
+
chunk = InformationChunk.new(label: label, chunk_type: chunk_type,
|
|
18
|
+
original_size: original_size)
|
|
19
|
+
@chunks[chunk.id] = chunk
|
|
20
|
+
chunk
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def compress_chunk(chunk_id:, amount: COMPRESSION_RATE)
|
|
24
|
+
chunk = @chunks[chunk_id]
|
|
25
|
+
return nil unless chunk
|
|
26
|
+
|
|
27
|
+
chunk.compress!(amount: amount)
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def decompress_chunk(chunk_id:, amount: COMPRESSION_RATE)
|
|
31
|
+
chunk = @chunks[chunk_id]
|
|
32
|
+
return nil unless chunk
|
|
33
|
+
|
|
34
|
+
chunk.decompress!(amount: amount)
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def abstract_chunks(chunk_ids:, abstraction_label:)
|
|
38
|
+
prune_abstractions_if_needed
|
|
39
|
+
chunks = chunk_ids.filter_map { |id| @chunks[id] }
|
|
40
|
+
return nil if chunks.empty?
|
|
41
|
+
|
|
42
|
+
combined_size = chunks.sum(&:compressed_size)
|
|
43
|
+
abstraction = InformationChunk.new(
|
|
44
|
+
label: abstraction_label,
|
|
45
|
+
chunk_type: :abstract,
|
|
46
|
+
original_size: combined_size
|
|
47
|
+
)
|
|
48
|
+
avg_fidelity = (chunks.sum(&:fidelity) / chunks.size).round(10)
|
|
49
|
+
abstraction.compress!(amount: DEFAULT_COMPRESSION_RATIO)
|
|
50
|
+
@abstractions[abstraction.id] = {
|
|
51
|
+
chunk: abstraction,
|
|
52
|
+
source_ids: chunk_ids,
|
|
53
|
+
fidelity: avg_fidelity
|
|
54
|
+
}
|
|
55
|
+
abstraction
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def compress_all(amount: COMPRESSION_RATE)
|
|
59
|
+
@chunks.each_value { |c| c.compress!(amount: amount) }
|
|
60
|
+
@chunks.size
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def chunks_by_type(chunk_type:)
|
|
64
|
+
ct = chunk_type.to_sym
|
|
65
|
+
@chunks.values.select { |c| c.chunk_type == ct }
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def highly_compressed_chunks
|
|
69
|
+
@chunks.values.select(&:highly_compressed?)
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
def average_compression_ratio
|
|
73
|
+
return 0.0 if @chunks.empty?
|
|
74
|
+
|
|
75
|
+
ratios = @chunks.values.map(&:compression_ratio)
|
|
76
|
+
(ratios.sum / ratios.size).round(10)
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def average_fidelity
|
|
80
|
+
return 1.0 if @chunks.empty?
|
|
81
|
+
|
|
82
|
+
fids = @chunks.values.map(&:fidelity)
|
|
83
|
+
(fids.sum / fids.size).round(10)
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
def total_original_size
|
|
87
|
+
@chunks.values.sum(&:original_size).round(10)
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def total_compressed_size
|
|
91
|
+
@chunks.values.sum(&:compressed_size).round(10)
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def overall_compression_ratio
|
|
95
|
+
return 0.0 if total_original_size.zero?
|
|
96
|
+
|
|
97
|
+
(1.0 - (total_compressed_size / total_original_size)).clamp(0.0, 1.0).round(10)
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def compression_report
|
|
101
|
+
{
|
|
102
|
+
total_chunks: @chunks.size,
|
|
103
|
+
total_abstractions: @abstractions.size,
|
|
104
|
+
average_compression_ratio: average_compression_ratio,
|
|
105
|
+
overall_compression_ratio: overall_compression_ratio,
|
|
106
|
+
average_fidelity: average_fidelity,
|
|
107
|
+
total_original_size: total_original_size,
|
|
108
|
+
total_compressed_size: total_compressed_size,
|
|
109
|
+
highly_compressed_count: highly_compressed_chunks.size
|
|
110
|
+
}
|
|
111
|
+
end
|
|
112
|
+
|
|
113
|
+
def to_h
|
|
114
|
+
{
|
|
115
|
+
total_chunks: @chunks.size,
|
|
116
|
+
total_abstractions: @abstractions.size,
|
|
117
|
+
average_compression_ratio: average_compression_ratio,
|
|
118
|
+
average_fidelity: average_fidelity
|
|
119
|
+
}
|
|
120
|
+
end
|
|
121
|
+
|
|
122
|
+
private
|
|
123
|
+
|
|
124
|
+
def prune_chunks_if_needed
|
|
125
|
+
return if @chunks.size < MAX_CHUNKS
|
|
126
|
+
|
|
127
|
+
oldest = @chunks.values.min_by(&:created_at)
|
|
128
|
+
@chunks.delete(oldest.id) if oldest
|
|
129
|
+
end
|
|
130
|
+
|
|
131
|
+
def prune_abstractions_if_needed
|
|
132
|
+
return if @abstractions.size < MAX_ABSTRACTIONS
|
|
133
|
+
|
|
134
|
+
oldest_key = @abstractions.min_by { |_, v| v[:chunk].created_at }&.first
|
|
135
|
+
@abstractions.delete(oldest_key) if oldest_key
|
|
136
|
+
end
|
|
137
|
+
end
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
end
|
|
141
|
+
end
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Legion
|
|
4
|
+
module Extensions
|
|
5
|
+
module CognitiveCompression
|
|
6
|
+
module Helpers
|
|
7
|
+
module Constants
|
|
8
|
+
MAX_CHUNKS = 500
|
|
9
|
+
MAX_ABSTRACTIONS = 200
|
|
10
|
+
|
|
11
|
+
DEFAULT_COMPRESSION_RATIO = 0.5
|
|
12
|
+
COMPRESSION_RATE = 0.1
|
|
13
|
+
FIDELITY_LOSS_RATE = 0.02
|
|
14
|
+
MIN_FIDELITY = 0.1
|
|
15
|
+
|
|
16
|
+
COMPRESSION_LABELS = {
|
|
17
|
+
(0.8..) => :highly_compressed,
|
|
18
|
+
(0.6...0.8) => :compressed,
|
|
19
|
+
(0.4...0.6) => :moderate,
|
|
20
|
+
(0.2...0.4) => :detailed,
|
|
21
|
+
(..0.2) => :raw
|
|
22
|
+
}.freeze
|
|
23
|
+
|
|
24
|
+
FIDELITY_LABELS = {
|
|
25
|
+
(0.8..) => :pristine,
|
|
26
|
+
(0.6...0.8) => :faithful,
|
|
27
|
+
(0.4...0.6) => :approximate,
|
|
28
|
+
(0.2...0.4) => :lossy,
|
|
29
|
+
(..0.2) => :degraded
|
|
30
|
+
}.freeze
|
|
31
|
+
|
|
32
|
+
CHUNK_TYPES = %i[
|
|
33
|
+
episodic semantic procedural sensory
|
|
34
|
+
emotional abstract relational
|
|
35
|
+
].freeze
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
end
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'securerandom'
|
|
4
|
+
|
|
5
|
+
module Legion
|
|
6
|
+
module Extensions
|
|
7
|
+
module CognitiveCompression
|
|
8
|
+
module Helpers
|
|
9
|
+
class InformationChunk
|
|
10
|
+
include Constants
|
|
11
|
+
|
|
12
|
+
attr_reader :id, :label, :chunk_type, :original_size, :compressed_size,
|
|
13
|
+
:fidelity, :compression_count, :created_at
|
|
14
|
+
|
|
15
|
+
def initialize(label:, chunk_type: :semantic, original_size: 1.0)
|
|
16
|
+
@id = SecureRandom.uuid
|
|
17
|
+
@label = label
|
|
18
|
+
@chunk_type = chunk_type.to_sym
|
|
19
|
+
@original_size = [original_size.to_f, 0.01].max
|
|
20
|
+
@compressed_size = @original_size
|
|
21
|
+
@fidelity = 1.0
|
|
22
|
+
@compression_count = 0
|
|
23
|
+
@created_at = Time.now.utc
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def compression_ratio
|
|
27
|
+
return 0.0 if @original_size.zero?
|
|
28
|
+
|
|
29
|
+
(1.0 - (@compressed_size / @original_size)).clamp(0.0, 1.0).round(10)
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def compress!(amount: COMPRESSION_RATE)
|
|
33
|
+
@compressed_size = (@compressed_size * (1.0 - amount)).clamp(0.01, @original_size).round(10)
|
|
34
|
+
@fidelity = (@fidelity - FIDELITY_LOSS_RATE).clamp(MIN_FIDELITY, 1.0).round(10)
|
|
35
|
+
@compression_count += 1
|
|
36
|
+
self
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def decompress!(amount: COMPRESSION_RATE)
|
|
40
|
+
@compressed_size = (@compressed_size / (1.0 - amount)).clamp(0.01, @original_size).round(10)
|
|
41
|
+
self
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def highly_compressed?
|
|
45
|
+
compression_ratio >= 0.8
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def compression_label
|
|
49
|
+
match = COMPRESSION_LABELS.find { |range, _| range.cover?(compression_ratio) }
|
|
50
|
+
match ? match.last : :raw
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def fidelity_label
|
|
54
|
+
match = FIDELITY_LABELS.find { |range, _| range.cover?(@fidelity) }
|
|
55
|
+
match ? match.last : :degraded
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def to_h
|
|
59
|
+
{
|
|
60
|
+
id: @id,
|
|
61
|
+
label: @label,
|
|
62
|
+
chunk_type: @chunk_type,
|
|
63
|
+
original_size: @original_size,
|
|
64
|
+
compressed_size: @compressed_size,
|
|
65
|
+
compression_ratio: compression_ratio,
|
|
66
|
+
compression_label: compression_label,
|
|
67
|
+
fidelity: @fidelity,
|
|
68
|
+
fidelity_label: fidelity_label,
|
|
69
|
+
compression_count: @compression_count,
|
|
70
|
+
created_at: @created_at
|
|
71
|
+
}
|
|
72
|
+
end
|
|
73
|
+
end
|
|
74
|
+
end
|
|
75
|
+
end
|
|
76
|
+
end
|
|
77
|
+
end
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Legion
|
|
4
|
+
module Extensions
|
|
5
|
+
module CognitiveCompression
|
|
6
|
+
module Runners
|
|
7
|
+
module CognitiveCompression
|
|
8
|
+
include Helpers::Constants
|
|
9
|
+
|
|
10
|
+
include Legion::Extensions::Helpers::Lex if defined?(Legion::Extensions::Helpers::Lex)
|
|
11
|
+
|
|
12
|
+
def store_chunk(label:, engine: nil, chunk_type: :semantic, original_size: 1.0, **)
|
|
13
|
+
eng = engine || default_engine
|
|
14
|
+
chunk = eng.store_chunk(label: label, chunk_type: chunk_type, original_size: original_size)
|
|
15
|
+
{ success: true, chunk: chunk.to_h }
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def compress_chunk(chunk_id:, engine: nil, amount: COMPRESSION_RATE, **)
|
|
19
|
+
eng = engine || default_engine
|
|
20
|
+
result = eng.compress_chunk(chunk_id: chunk_id, amount: amount)
|
|
21
|
+
return { success: false, error: 'chunk not found' } unless result
|
|
22
|
+
|
|
23
|
+
{ success: true, chunk: result.to_h }
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def decompress_chunk(chunk_id:, engine: nil, amount: COMPRESSION_RATE, **)
|
|
27
|
+
eng = engine || default_engine
|
|
28
|
+
result = eng.decompress_chunk(chunk_id: chunk_id, amount: amount)
|
|
29
|
+
return { success: false, error: 'chunk not found' } unless result
|
|
30
|
+
|
|
31
|
+
{ success: true, chunk: result.to_h }
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def abstract_chunks(chunk_ids:, abstraction_label:, engine: nil, **)
|
|
35
|
+
eng = engine || default_engine
|
|
36
|
+
result = eng.abstract_chunks(chunk_ids: chunk_ids, abstraction_label: abstraction_label)
|
|
37
|
+
return { success: false, error: 'no valid chunks' } unless result
|
|
38
|
+
|
|
39
|
+
{ success: true, abstraction: result.to_h }
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def compress_all(engine: nil, amount: COMPRESSION_RATE, **)
|
|
43
|
+
eng = engine || default_engine
|
|
44
|
+
count = eng.compress_all(amount: amount)
|
|
45
|
+
{ success: true, compressed_count: count }
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def average_fidelity(engine: nil, **)
|
|
49
|
+
eng = engine || default_engine
|
|
50
|
+
{ success: true, fidelity: eng.average_fidelity }
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def overall_compression_ratio(engine: nil, **)
|
|
54
|
+
eng = engine || default_engine
|
|
55
|
+
{ success: true, ratio: eng.overall_compression_ratio }
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def compression_report(engine: nil, **)
|
|
59
|
+
eng = engine || default_engine
|
|
60
|
+
{ success: true, report: eng.compression_report }
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
private
|
|
64
|
+
|
|
65
|
+
def default_engine
|
|
66
|
+
@default_engine ||= Helpers::CompressionEngine.new
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|
|
71
|
+
end
|
|
72
|
+
end
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative 'cognitive_compression/version'
|
|
4
|
+
require_relative 'cognitive_compression/helpers/constants'
|
|
5
|
+
require_relative 'cognitive_compression/helpers/information_chunk'
|
|
6
|
+
require_relative 'cognitive_compression/helpers/compression_engine'
|
|
7
|
+
require_relative 'cognitive_compression/runners/cognitive_compression'
|
|
8
|
+
require_relative 'cognitive_compression/client'
|
|
9
|
+
|
|
10
|
+
module Legion
|
|
11
|
+
module Extensions
|
|
12
|
+
module CognitiveCompression
|
|
13
|
+
end
|
|
14
|
+
end
|
|
15
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: lex-cognitive-compression
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Esity
|
|
8
|
+
bindir: bin
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies:
|
|
12
|
+
- !ruby/object:Gem::Dependency
|
|
13
|
+
name: legion-gaia
|
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
|
15
|
+
requirements:
|
|
16
|
+
- - ">="
|
|
17
|
+
- !ruby/object:Gem::Version
|
|
18
|
+
version: '0'
|
|
19
|
+
type: :development
|
|
20
|
+
prerelease: false
|
|
21
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
22
|
+
requirements:
|
|
23
|
+
- - ">="
|
|
24
|
+
- !ruby/object:Gem::Version
|
|
25
|
+
version: '0'
|
|
26
|
+
description: Models information compression and abstraction as knowledge moves through
|
|
27
|
+
memory tiers
|
|
28
|
+
email:
|
|
29
|
+
- matthewdiverson@gmail.com
|
|
30
|
+
executables: []
|
|
31
|
+
extensions: []
|
|
32
|
+
extra_rdoc_files: []
|
|
33
|
+
files:
|
|
34
|
+
- ".github/workflows/ci.yml"
|
|
35
|
+
- ".gitignore"
|
|
36
|
+
- ".rspec"
|
|
37
|
+
- ".rubocop.yml"
|
|
38
|
+
- CLAUDE.md
|
|
39
|
+
- Gemfile
|
|
40
|
+
- Gemfile.lock
|
|
41
|
+
- README.md
|
|
42
|
+
- lex-cognitive-compression.gemspec
|
|
43
|
+
- lib/legion/extensions/cognitive_compression.rb
|
|
44
|
+
- lib/legion/extensions/cognitive_compression/client.rb
|
|
45
|
+
- lib/legion/extensions/cognitive_compression/helpers/compression_engine.rb
|
|
46
|
+
- lib/legion/extensions/cognitive_compression/helpers/constants.rb
|
|
47
|
+
- lib/legion/extensions/cognitive_compression/helpers/information_chunk.rb
|
|
48
|
+
- lib/legion/extensions/cognitive_compression/runners/cognitive_compression.rb
|
|
49
|
+
- lib/legion/extensions/cognitive_compression/version.rb
|
|
50
|
+
homepage: https://github.com/LegionIO/lex-cognitive-compression
|
|
51
|
+
licenses:
|
|
52
|
+
- MIT
|
|
53
|
+
metadata:
|
|
54
|
+
homepage_uri: https://github.com/LegionIO/lex-cognitive-compression
|
|
55
|
+
source_code_uri: https://github.com/LegionIO/lex-cognitive-compression
|
|
56
|
+
documentation_uri: https://github.com/LegionIO/lex-cognitive-compression/blob/main/README.md
|
|
57
|
+
changelog_uri: https://github.com/LegionIO/lex-cognitive-compression/blob/main/CHANGELOG.md
|
|
58
|
+
bug_tracker_uri: https://github.com/LegionIO/lex-cognitive-compression/issues
|
|
59
|
+
rubygems_mfa_required: 'true'
|
|
60
|
+
rdoc_options: []
|
|
61
|
+
require_paths:
|
|
62
|
+
- lib
|
|
63
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
64
|
+
requirements:
|
|
65
|
+
- - ">="
|
|
66
|
+
- !ruby/object:Gem::Version
|
|
67
|
+
version: '3.4'
|
|
68
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
69
|
+
requirements:
|
|
70
|
+
- - ">="
|
|
71
|
+
- !ruby/object:Gem::Version
|
|
72
|
+
version: '0'
|
|
73
|
+
requirements: []
|
|
74
|
+
rubygems_version: 3.6.9
|
|
75
|
+
specification_version: 4
|
|
76
|
+
summary: Cognitive information compression for LegionIO
|
|
77
|
+
test_files: []
|