marco-dvcs 0.1.0__tar.gz

marco_dvcs-0.1.0/PKG-INFO

@@ -0,0 +1,123 @@
+Metadata-Version: 2.4
+Name: marco-dvcs
+Version: 0.1.0
+Summary: A minimal dataset versioning system for text data with a focus on reproducibility.
+Home-page: https://github.com/Team-Marco-ACM/marco-package
+Author: Your Name
+Author-email: your.email@example.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: Flask
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Marco Dataset Versioning System
+
+A minimal dataset versioning system for text data with a strong focus on reproducibility and transparency. Treat your text datasets like code — immutable, versioned, reproducible, and explainable.
+
+Marco acts as a lightweight Python library, meaning you can initialize it in *any* machine learning project folder to safely version and preprocess your datasets without altering your original files.
+
+## 🚀 Installation (Linux / MacOS)
+
+On modern Linux environments (like Arch Linux, Ubuntu 23.04+), Python packages must be installed in a Virtual Environment (PEP 668) to prevent conflicts with your system packages. 
+
+Follow these steps to safely install Marco into your ML project:
+
+1. **Clone this repository** to your local machine:
+   ```bash
+   git clone https://github.com/your-username/marco.git
+   cd marco
+   ```
+
+2. **Navigate to the ML project folder** where you want to train your model (e.g. your bag-of-words project):
+   ```bash
+   cd ~/projects/my-bag-of-words-model
+   ```
+
+3. **Create and activate a Python Virtual Environment**:
+   ```bash
+   # Create a virtual environment named 'venv'
+   python3 -m venv venv
+   
+   # Activate it (You must do this every time you open a new terminal in this folder)
+   source venv/bin/activate
+   ```
+   *(You should now see `(venv)` at the start of your terminal prompt!)*
+
+4. **Install Marco**:
+   ```bash
+   # Point pip to the directory where you cloned the marco repository
+   pip install -e /path/to/marco
+   ```
+
+---
+
+## 🛠️ Usage Guide
+
+Once `marco` is installed in your virtual environment, you have access to the full CLI!
+
+### 1. Initialize a Repository
+Initialize Marco tracking in your current directory. This creates a `.marco/` data versioning environment specific to that project.
+```bash
+marco init
+```
+
+### 2. Create an Immutable Version 
+Upload a text/CSV/TSV dataset to create an immutable version. Marco will compute a cryptographically secure SHA-256 hash using the raw data + the preprocessing configuration.
+
+**Interactive Mode:**
+If you don't supply a configuration file, Marco will interactively guide you through building the preprocessing pipeline (Lowercasing, Tokenization, Stopwords Removal, Deduplicating).
+```bash
+marco upload my_dataset.csv -t v1-raw
+```
+
+**Config Mode:**
+```bash
+marco upload my_dataset.csv -c my_config.json -t v1-processed
+```
+
+### 3. List Versions
+View all the versions you've created, along with their tags and timestamps.
+```bash
+marco list
+```
+
+### 4. Start the Visual Dashboard
+Open the Flask-powered web dashboard to view dataset metadata, differences between versions, and complete data lineage trees.
+```bash
+python -m marco.web.app --port 5000
+```
+Open `http://localhost:5000` in your web browser.
+
+### 5. Export / Import Versions
+Easily share dataset versions with teammates by packing them into `.tar.gz` files.
+```bash
+# Export version 'v1-raw' to the 'exports' folder
+marco export v1-raw ./exports/
+
+# Import an archive sent to you by a coworker
+marco import ./exports/marco_version_e5e0b767.tar.gz
+```
+
+---
+
+## 🧠 Architecture Overview
+
+Marco decouples logic from the file system. All core engine operations sit inside `marco/core/`, including:
+- `locker.py`: File-based concurrency control using `.lock` files.
+- `repository.py`: CRUD operations for dataset versions and `refs.json` tagging.
+- `preprocessor.py`: A robust Directed Acyclic Graph (DAG) preprocessing engine.
+
+Have fun building safer machine learning pipelines!

marco_dvcs-0.1.0/README.md

@@ -0,0 +1,98 @@
+# Marco Dataset Versioning System
+
+A minimal dataset versioning system for text data with a strong focus on reproducibility and transparency. Treat your text datasets like code — immutable, versioned, reproducible, and explainable.
+
+Marco acts as a lightweight Python library, meaning you can initialize it in *any* machine learning project folder to safely version and preprocess your datasets without altering your original files.
+
+## 🚀 Installation (Linux / MacOS)
+
+On modern Linux environments (like Arch Linux, Ubuntu 23.04+), Python packages must be installed in a Virtual Environment (PEP 668) to prevent conflicts with your system packages. 
+
+Follow these steps to safely install Marco into your ML project:
+
+1. **Clone this repository** to your local machine:
+   ```bash
+   git clone https://github.com/your-username/marco.git
+   cd marco
+   ```
+
+2. **Navigate to the ML project folder** where you want to train your model (e.g. your bag-of-words project):
+   ```bash
+   cd ~/projects/my-bag-of-words-model
+   ```
+
+3. **Create and activate a Python Virtual Environment**:
+   ```bash
+   # Create a virtual environment named 'venv'
+   python3 -m venv venv
+   
+   # Activate it (You must do this every time you open a new terminal in this folder)
+   source venv/bin/activate
+   ```
+   *(You should now see `(venv)` at the start of your terminal prompt!)*
+
+4. **Install Marco**:
+   ```bash
+   # Point pip to the directory where you cloned the marco repository
+   pip install -e /path/to/marco
+   ```
+
+---
+
+## 🛠️ Usage Guide
+
+Once `marco` is installed in your virtual environment, you have access to the full CLI!
+
+### 1. Initialize a Repository
+Initialize Marco tracking in your current directory. This creates a `.marco/` data versioning environment specific to that project.
+```bash
+marco init
+```
+
+### 2. Create an Immutable Version 
+Upload a text/CSV/TSV dataset to create an immutable version. Marco will compute a cryptographically secure SHA-256 hash using the raw data + the preprocessing configuration.
+
+**Interactive Mode:**
+If you don't supply a configuration file, Marco will interactively guide you through building the preprocessing pipeline (Lowercasing, Tokenization, Stopwords Removal, Deduplicating).
+```bash
+marco upload my_dataset.csv -t v1-raw
+```
+
+**Config Mode:**
+```bash
+marco upload my_dataset.csv -c my_config.json -t v1-processed
+```
+
+### 3. List Versions
+View all the versions you've created, along with their tags and timestamps.
+```bash
+marco list
+```
+
+### 4. Start the Visual Dashboard
+Open the Flask-powered web dashboard to view dataset metadata, differences between versions, and complete data lineage trees.
+```bash
+python -m marco.web.app --port 5000
+```
+Open `http://localhost:5000` in your web browser.
+
+### 5. Export / Import Versions
+Easily share dataset versions with teammates by packing them into `.tar.gz` files.
+```bash
+# Export version 'v1-raw' to the 'exports' folder
+marco export v1-raw ./exports/
+
+# Import an archive sent to you by a coworker
+marco import ./exports/marco_version_e5e0b767.tar.gz
+```
+
+---
+
+## 🧠 Architecture Overview
+
+Marco decouples logic from the file system. All core engine operations sit inside `marco/core/`, including:
+- `locker.py`: File-based concurrency control using `.lock` files.
+- `repository.py`: CRUD operations for dataset versions and `refs.json` tagging.
+- `preprocessor.py`: A robust Directed Acyclic Graph (DAG) preprocessing engine.
+
+Have fun building safer machine learning pipelines!

marco_dvcs-0.1.0/marco/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

marco_dvcs-0.1.0/marco/__main__.py

@@ -0,0 +1,4 @@
+from marco.cli.main import main
+
+if __name__ == "__main__":
+    main()

marco_dvcs-0.1.0/marco/cli/interactive.py

@@ -0,0 +1,41 @@
+import json
+
+def build_interactive_config() -> dict:
+    print("Welcome to Marco Interactive Dataset Builder!")
+    pipeline = []
+    
+    if input("1. Clean newlines? (Y/n): ").strip().lower() != 'n':
+        pipeline.append({"func": "normalize_newlines", "params": {}, "depends_on": []})
+        
+    if input("2. Convert to lowercase? (Y/n): ").strip().lower() != 'n':
+        pipeline.append({"func": "lowercase", "params": {"enabled": True}, "depends_on": []})
+        
+    if input("3. Tokenize? (Y/n): ").strip().lower() != 'n':
+        method = input("  Method [whitespace]: ").strip() or "whitespace"
+        pipeline.append({"func": "tokenize", "params": {"method": method}, "depends_on": []})
+        
+    if input("4. Remove Stopwords? (Y/n): ").strip().lower() != 'n':
+        lang = input("  Language [english]: ").strip() or "english"
+        pipeline.append({"func": "remove_stopwords", "params": {"language": lang}, "depends_on": []})
+
+    if input("5. Filter by length? (Y/n): ").strip().lower() != 'n':
+        min_t = input("  Min tokens [1]: ").strip() or "1"
+        max_t = input("  Max tokens [1000]: ").strip() or "1000"
+        pipeline.append({"func": "filter_length", "params": {"min_tokens": int(min_t), "max_tokens": int(max_t)}, "depends_on": []})
+        
+    if input("6. Deduplicate? (Y/n): ").strip().lower() != 'n':
+        pipeline.append({"func": "deduplicate", "params": {"method": "exact"}, "depends_on": []})
+
+    dag = {}
+    prev = None
+    for i, step in enumerate(pipeline):
+        step_name = f"step_{i+1}_{step['func']}"
+        if prev:
+            step["depends_on"] = [prev]
+        dag[step_name] = step
+        prev = step_name
+        
+    print("\nGenerated Pipeline Configuration:")
+    print(json.dumps({"dag": dag}, indent=2))
+    
+    return {"dag": dag}

marco_dvcs-0.1.0/marco/cli/main.py

@@ -0,0 +1,76 @@
+import argparse
+import sys
+import json
+from pathlib import Path
+from marco.core import repository
+from marco.cli.interactive import build_interactive_config
+
+def main():
+    parser = argparse.ArgumentParser(prog="marco", description="Marco Dataset Versioning System")
+    subparsers = parser.add_subparsers(dest="command", required=True)
+    
+    p_init = subparsers.add_parser("init", help="Initialize a marco repository")
+    p_init.add_argument("path", nargs="?", default=".", help="Path to initialize")
+    
+    p_upload = subparsers.add_parser("upload", help="Create a dataset version")
+    p_upload.add_argument("file", help="Path to raw dataset string/CSV/TSV")
+    p_upload.add_argument("-c", "--config", help="Path to config.json")
+    p_upload.add_argument("-t", "--tag", help="Tag for the version")
+    p_upload.add_argument("-u", "--user", default="unknown", help="User name")
+    
+    subparsers.add_parser("list", help="List all dataset versions")
+    
+    p_export = subparsers.add_parser("export", help="Export a version to tar.gz")
+    p_export.add_argument("version", help="Version Hash or Tag")
+    p_export.add_argument("dest", help="Destination folder")
+    
+    p_import = subparsers.add_parser("import", help="Import a version from tar.gz")
+    p_import.add_argument("tarball", help="Path to exported tar.gz")
+    
+    args = parser.parse_args()
+    repo_path = Path.cwd()
+    
+    if args.command == "init":
+        p = args.path if args.path else "."
+        repository.init_repo(p)
+        print(f"Initialized Marco repository in {Path(p).resolve() / '.marco'}")
+        
+    elif args.command == "upload":
+        raw_path = Path(args.file)
+        if not raw_path.exists():
+            print(f"Error: File {raw_path} does not exist.")
+            sys.exit(1)
+            
+        if args.config:
+            config = json.loads(Path(args.config).read_text(encoding='utf-8'))
+        else:
+            config = build_interactive_config()
+            
+        tags = [args.tag] if args.tag else []
+        vid = repository.create_version(raw_path, config, repo_path, user=args.user, tags=tags)
+        print(f"Successfully processed and generated version: {vid}")
+        
+    elif args.command == "list":
+        try:
+            versions = repository.list_versions(repo_path)
+            print(f"{'Version':<10} | {'Created At':<20} | {'User':<10} | {'Tags':<20}")
+            print("-" * 68)
+            for v in versions:
+                short = v['version_id'][:8]
+                d = v.get('created_at', '')[:19].replace('T', ' ')
+                u = v.get('created_by', 'unknown')
+                t = ", ".join(v.get('tags', []))
+                print(f"{short:<10} | {d:<20} | {u:<10} | {t:<20}")
+        except Exception as e:
+            print(f"Failed to list versions: {e}")
+            
+    elif args.command == "export":
+        tar_path = repository.export_version(args.version, args.dest, repo_path)
+        print(f"Exported to {tar_path}")
+        
+    elif args.command == "import":
+        vid = repository.import_version(args.tarball, repo_path)
+        print(f"Successfully imported version {vid}")
+
+if __name__ == "__main__":
+    main()

marco_dvcs-0.1.0/marco/cli/web_serve.py

@@ -0,0 +1,7 @@
+# This file is reserved for Developer 2 to implement the React Web Server logic.
+# The `marco-generate-web` CLI command will point here.
+
+def serve_react_app():
+    print("React visualization dashboard coming soon!")
+    # TODO: Implement local HTTP server for the compiled React App in marco/ui/dist
+    pass

marco_dvcs-0.1.0/marco/core/locker.py

@@ -0,0 +1,42 @@
+import time
+import os
+import contextlib
+from pathlib import Path
+
+class LockAcquisitionError(Exception):
+    pass
+
+@contextlib.contextmanager
+def FileLock(lock_dir: Path, name: str, timeout: int = 10, backoff: float = 0.5):
+    """
+    A file-based lock using atomic file creation mode ('x').
+    Retries with an exponential-like or constant backoff until timeout.
+    """
+    lock_dir = Path(lock_dir)
+    lock_dir.mkdir(parents=True, exist_ok=True)
+    lock_file = lock_dir / f"{name}.lock"
+    
+    start_time = time.time()
+    acquired = False
+    
+    while time.time() - start_time < timeout:
+        try:
+            # Atomic creation. Fails if file already exists.
+            with open(lock_file, "x") as f:
+                f.write(str(os.getpid()))
+            acquired = True
+            break
+        except FileExistsError:
+            time.sleep(backoff)
+            
+    if not acquired:
+        raise LockAcquisitionError(f"Could not acquire lock '{name}' within {timeout} seconds")
+        
+    try:
+        yield
+    finally:
+        try:
+            if lock_file.exists():
+                lock_file.unlink()
+        except OSError:
+            pass

marco_dvcs-0.1.0/marco/core/preprocessor.py

@@ -0,0 +1,268 @@
+# scripts/pipeline_dag.py
+import json
+import hashlib
+from collections import defaultdict, deque
+from typing import List, Tuple, Dict, Callable, Any
+
+# allowed libs - used elsewhere
+import pandas as pd
+
+# Type: data is list of (label, text)
+DataRows = List[Tuple[str, str]]
+NodeSpec = Dict[str, Any]
+PipelineSpec = Dict[str, NodeSpec]
+
+ENGLISH_STOPWORDS = {
+    "a", "about", "above", "after", "again", "against", "all", "am", "an", "and", "any", "are", "aren't", "as", "at",
+    "be", "because", "been", "before", "being", "below", "between", "both", "but", "by", "can't", "cannot", "could",
+    "couldn't", "did", "didn't", "do", "does", "doesn't", "doing", "don't", "down", "during", "each", "few", "for",
+    "from", "further", "had", "hadn't", "has", "hasn't", "have", "haven't", "having", "he", "he'd", "he'll", "he's",
+    "her", "here", "here's", "hers", "herself", "him", "himself", "his", "how", "how's", "i", "i'd", "i'll", "i'm",
+    "i've", "if", "in", "into", "is", "isn't", "it", "it's", "its", "itself", "let's", "me", "more", "most", "mustn't",
+    "my", "myself", "no", "nor", "not", "of", "off", "on", "once", "only", "or", "other", "ought", "our", "ours",
+    "ourselves", "out", "over", "own", "same", "shan't", "she", "she'd", "she'll", "she's", "should", "shouldn't", "so",
+    "some", "such", "than", "that", "that's", "the", "their", "theirs", "them", "themselves", "then", "there", "there's",
+    "these", "they", "they'd", "they'll", "they're", "they've", "this", "those", "through", "to", "too", "under", "until",
+    "up", "very", "was", "wasn't", "we", "we'd", "we'll", "we're", "we've", "were", "weren't", "what", "what's", "when",
+    "when's", "where", "where's", "which", "while", "who", "who's", "whom", "why", "why's", "with", "won't", "would",
+    "wouldn't", "you", "you'd", "you'll", "you're", "you've", "your", "yours", "yourself", "yourselves"
+}
+
+# -------------------------
+# Node implementations
+# -------------------------
+# Each node function takes DataRows and params, returns DataRows.
+# Implementations MUST be deterministic.
+
+def normalize_newlines_node(data: DataRows, params: dict) -> DataRows:
+    out = []
+    for lbl, txt in data:
+        t = txt.replace('\r\n', '\n').replace('\r', '\n')
+        out.append((lbl, t))
+    return out
+
+def lowercase_node(data: DataRows, params: dict) -> DataRows:
+    enabled = params.get('enabled', True)
+    if not enabled:
+        return data
+    return [(lbl, txt.lower()) for lbl, txt in data]
+
+def uppercase_custom_node(data: DataRows, params: dict) -> DataRows:
+    return [(lbl, txt.upper()) for lbl, txt in data]
+
+def tokenize_node(data: DataRows, params: dict) -> DataRows:
+    # Deterministic whitespace tokenization; store tokens as space-joined text
+    method = params.get('method', 'whitespace')
+    out = []
+    for lbl, txt in data:
+        if method == 'whitespace':
+            tokens = [t for t in txt.strip().split() if t]
+        else:
+            # fallback: whitespace
+            tokens = [t for t in txt.strip().split() if t]
+        out.append((lbl, ' '.join(tokens)))
+    return out
+
+def filter_length_node(data: DataRows, params: dict) -> DataRows:
+    min_t = int(params.get('min_tokens', 0))
+    max_t = int(params.get('max_tokens', 10_000_000))
+    out = []
+    for lbl, txt in data:
+        n = len(txt.split()) if txt else 0
+        if min_t <= n <= max_t:
+            out.append((lbl, txt))
+    return out
+
+def remove_stopwords_node(data: DataRows, params: dict) -> DataRows:
+    language = params.get('language', 'english')
+    # only support english built-in for now
+    if language != 'english':
+        return data
+        
+    out = []
+    for lbl, txt in data:
+        tokens = txt.split()
+        filtered = [t for t in tokens if t.lower() not in ENGLISH_STOPWORDS]
+        out.append((lbl, ' '.join(filtered)))
+    return out
+
+def deduplicate_node(data: DataRows, params: dict) -> DataRows:
+    # Keep first occurrence deterministically
+    seen = set()
+    out = []
+    method = params.get('method', 'exact')
+    for lbl, txt in data:
+        key = txt if method == 'exact' else hashlib.sha256(txt.encode('utf-8')).hexdigest()
+        if key in seen:
+            continue
+        seen.add(key)
+        out.append((lbl, txt))
+    return out
+
+# node registry maps string name to function
+_NODE_REGISTRY: Dict[str, Callable[[DataRows, dict], DataRows]] = {
+    'normalize_newlines': normalize_newlines_node,
+    'lowercase': lowercase_node,
+    'uppercase_custom': uppercase_custom_node,
+    'tokenize': tokenize_node,
+    'filter_length': filter_length_node,
+    'remove_stopwords': remove_stopwords_node,
+    'deduplicate': deduplicate_node,
+}
+
+# -------------------------
+# DAG helpers
+# -------------------------
+def validate_dag(spec: PipelineSpec):
+    # Check for cycles using DFS detection
+    visited = {}
+    def dfs(node):
+        if node not in spec:
+            raise ValueError(f"Node '{node}' is declared as dependency but missing from pipeline spec")
+        if visited.get(node) == 'visiting':
+            raise ValueError(f"Cycle detected at node: {node}")
+        if visited.get(node) == 'visited':
+            return
+        visited[node] = 'visiting'
+        for dep in spec[node].get('depends_on', []):
+            dfs(dep)
+        visited[node] = 'visited'
+    for n in spec:
+        if n not in visited:
+            dfs(n)
+    return True
+
+def topological_sort_order(spec: PipelineSpec) -> List[str]:
+    # Kahn's algorithm (deterministic: sort nodes to break ties)
+    indeg = {n: 0 for n in spec}
+    deps = defaultdict(list)
+    for n, s in spec.items():
+        for d in s.get('depends_on', []):
+            indeg[n] += 1
+            deps[d].append(n)
+    # start with indeg 0 nodes sorted
+    q = deque(sorted([n for n, d in indeg.items() if d == 0]))
+    order = []
+    while q:
+        n = q.popleft()
+        order.append(n)
+        for child in sorted(deps.get(n, [])):
+            indeg[child] -= 1
+            if indeg[child] == 0:
+                q.append(child)
+    if len(order) != len(spec):
+        raise ValueError("Cycle or missing nodes detected in pipeline spec")
+    return order
+
+# -------------------------
+# Metrics helper
+# -------------------------
+def compute_metrics_for_rows(rows: DataRows) -> dict:
+    n_docs = len(rows)
+    total_tokens = 0
+    vocab = {}
+    n_empty = 0
+    for lbl, txt in rows:
+        tokens = txt.split() if txt else []
+        if not tokens:
+            n_empty += 1
+        total_tokens += len(tokens)
+        for t in tokens:
+            vocab[t] = vocab.get(t, 0) + 1
+    return {
+        'n_documents': int(n_docs),
+        'n_tokens': int(total_tokens),
+        'vocab_size': int(len(vocab)),
+        'avg_tokens_per_doc': float(total_tokens / n_docs) if n_docs else 0.0,
+        'median_tokens_per_doc': float(pd.Series([len(txt.split()) for _, txt in rows]).median()) if n_docs else 0.0,
+        'n_empty_docs': int(n_empty),
+        'top_tokens': [{'token': t, 'count': c} for t, c in sorted(vocab.items(), key=lambda x: -x[1])[:50]],
+    }
+
+# -------------------------
+# Runner
+# -------------------------
+def run_pipeline_dag(raw_rows: DataRows, pipeline_spec: PipelineSpec) -> (DataRows, dict, dict):
+    """
+    raw_rows: list of (label, text)
+    pipeline_spec: dict mapping node_name -> { 'func': <string key in registry>, 'params': {...}, 'depends_on': [...] }
+    Returns:
+      final_rows: DataRows (concatenation of final nodes' outputs)
+      metrics_per_step: dict mapping node_name -> metrics
+      dag_structure: minimal structure (nodes + depends_on)
+    """
+
+    # Validate and get order
+    validate_dag(pipeline_spec)
+    order = topological_sort_order(pipeline_spec)
+
+    results: Dict[str, DataRows] = {}
+    metrics_per_step = {}
+
+    # raw is the canonical input name
+    results['raw'] = raw_rows
+
+    # run nodes in deterministic order
+    for node in order:
+        node_spec = pipeline_spec[node]
+        func_key = node_spec.get('func')
+        params = node_spec.get('params', {})
+        depends = node_spec.get('depends_on', [])
+
+        # gather inputs deterministically: concat in order of depends list; if no deps use raw
+        if depends:
+            inputs = []
+            for d in depends:
+                # If dependency not run (shouldn't happen), raise
+                if d not in results:
+                    raise RuntimeError(f"Dependency output missing for {d}")
+                inputs.extend(results[d])
+        else:
+            inputs = results['raw']
+
+        # get function
+        if func_key not in _NODE_REGISTRY:
+            raise ValueError(f"Unknown node func '{func_key}' for node '{node}'")
+        func = _NODE_REGISTRY[func_key]
+        out_rows = func(inputs, params)
+
+        # store result and metrics
+        results[node] = out_rows
+        metrics_per_step[node] = compute_metrics_for_rows(out_rows)
+
+    # Determine final nodes: nodes that are not depended upon by any other node
+    depended = set()
+    for n, s in pipeline_spec.items():
+        for d in s.get('depends_on', []):
+            depended.add(d)
+    final_nodes = sorted([n for n in pipeline_spec.keys() if n not in depended])
+    # Deterministic final output: concat outputs of final nodes in sorted order
+    final_output: DataRows = []
+    if final_nodes:
+        for fn in final_nodes:
+            final_output.extend(results[fn])
+    else:
+        # fallback: last node in order
+        final_output = results[order[-1]]
+
+    dag_structure = {n: {'depends_on': pipeline_spec[n].get('depends_on', []), 'func': pipeline_spec[n].get('func'), 'params': pipeline_spec[n].get('params', {})} for n in pipeline_spec}
+
+    return final_output, metrics_per_step, dag_structure
+
+# -------------------------
+# Serialization helpers for canonical bytes (for output hash)
+# -------------------------
+def canonical_preprocessed_bytes_from_rows(rows: DataRows) -> bytes:
+    # deterministic columns: label, text, n_tokens
+    import io
+    import csv
+    buf = io.StringIO()
+    writer = csv.writer(buf, delimiter='\t', lineterminator='\n', quoting=csv.QUOTE_MINIMAL)
+    writer.writerow(['label', 'text', 'n_tokens'])
+    for lbl, txt in rows:
+        writer.writerow([lbl, txt, str(len(txt.split()) if txt else 0)])
+    return buf.getvalue().encode('utf-8')
+
+def compute_output_hash_from_rows(rows: DataRows) -> str:
+    b = canonical_preprocessed_bytes_from_rows(rows)
+    return hashlib.sha256(b).hexdigest()

marco_dvcs-0.1.0/marco/core/repository.py

@@ -0,0 +1,213 @@
+import hashlib
+import json
+import os
+import shutil
+import tempfile
+import tarfile
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import List, Dict, Union
+
+import pandas as pd
+
+from marco.core.preprocessor import run_pipeline_dag, compute_output_hash_from_rows, compute_metrics_for_rows
+from marco.core.locker import FileLock
+
+def init_repo(repo_path: Union[str, Path]):
+    repo_path = Path(repo_path)
+    marco_dir = repo_path / '.marco'
+    (marco_dir / 'versions').mkdir(parents=True, exist_ok=True)
+    (marco_dir / 'locks').mkdir(parents=True, exist_ok=True)
+    refs_file = marco_dir / 'refs.json'
+    if not refs_file.exists():
+        refs_file.write_text('{}', encoding='utf-8')
+    return marco_dir
+
+def canonical_json(obj) -> str:
+    return json.dumps(obj, sort_keys=True, separators=(',', ':'))
+
+def compute_raw_hash(raw_path: Path) -> str:
+    hasher = hashlib.sha256()
+    with raw_path.open('rb') as f:
+        for chunk in iter(lambda: f.read(8192), b''):
+            hasher.update(chunk.replace(b"\r\n", b"\n"))
+    return hasher.hexdigest()
+
+def create_version(raw_path: Path, config: dict, repo_path: Path, user: str = "unknown", tags: List[str] = None, parents: List[str] = None):
+    tags = tags or []
+    parents = parents or []
+    repo_path = Path(repo_path)
+    marco_dir = repo_path / '.marco'
+    versions_dir = marco_dir / 'versions'
+    locks_dir = marco_dir / 'locks'
+    
+    if not marco_dir.exists():
+        raise RuntimeError(f"Marco repository not initialized at {repo_path}. Run 'marco init' first.")
+        
+    raw_hash = compute_raw_hash(raw_path)
+    canonical = canonical_json(config)
+    version_id = hashlib.sha256((raw_hash + ':' + canonical).encode('utf-8')).hexdigest()
+    version_folder = versions_dir / version_id
+    
+    with FileLock(locks_dir, "create_version"):
+        if version_folder.exists():
+            print(f"Version {version_id} already exists.")
+            if tags:
+                tag_version(version_id, tags, repo_path)
+            return version_id
+            
+        text = raw_path.read_text(encoding='utf-8', errors='replace')
+        text = text.replace('\r\n', '\n').replace('\r', '\n')
+        lines = [ln for ln in text.split('\n') if ln.strip()]
+        
+        raw_rows = []
+        for ln in lines:
+            if '\t' in ln:
+                lbl, txt = ln.split('\t', 1)
+            else:
+                lbl, txt = '', ln
+            raw_rows.append((lbl.strip(), txt.strip()))
+
+        pipeline_spec = config.get('dag') or {}
+        if not pipeline_spec:
+            raise ValueError("Config must include 'dag' mapping of nodes")
+
+        final_rows, metrics_per_step, dag_structure = run_pipeline_dag(raw_rows, pipeline_spec)
+        
+        final_metrics = compute_metrics_for_rows(final_rows)
+        output_hash = compute_output_hash_from_rows(final_rows)
+        final_metrics['output_hash'] = output_hash
+        final_metrics['created_by'] = user
+        
+        df = pd.DataFrame([{'label': l, 'text': t, 'n_tokens': len(t.split()) if t else 0} for l, t in final_rows])
+        
+        manifest = {
+            'version_id': version_id,
+            'created_at': datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"),
+            'created_by': user,
+            'raw_hash': raw_hash,
+            'config_hash': hashlib.sha256(canonical.encode('utf-8')).hexdigest(),
+            'n_raw_lines': len(lines),
+            'parents': parents,
+            'output_hash': output_hash,
+            'dag': dag_structure
+        }
+        
+        tmp = Path(tempfile.mkdtemp(prefix='version_tmp_', dir=versions_dir))
+        try:
+            shutil.copy2(raw_path, tmp / 'raw.txt')
+            (tmp / 'config.json').write_text(canonical, encoding='utf-8')
+            (tmp / 'raw_hash.txt').write_text(raw_hash, encoding='utf-8')
+            df.to_csv(tmp / 'preprocessed.tsv', sep='\t', index=False, header=True, encoding='utf-8', lineterminator='\n')
+            final_metrics['metrics_per_step'] = metrics_per_step
+            (tmp / 'metrics.json').write_text(json.dumps(final_metrics, indent=2), encoding='utf-8')
+            (tmp / 'dag_structure.json').write_text(json.dumps(dag_structure, indent=2), encoding='utf-8')
+            (tmp / 'manifest.json').write_text(json.dumps(manifest, indent=2), encoding='utf-8')
+            
+            os.replace(str(tmp), str(version_folder))
+        except Exception:
+            shutil.rmtree(tmp, ignore_errors=True)
+            raise
+
+    if tags:
+        tag_version(version_id, tags, repo_path)
+        
+    return version_id
+
+def tag_version(version_id: str, tags: List[str], repo_path: Path):
+    repo_path = Path(repo_path)
+    refs_file = repo_path / '.marco' / 'refs.json'
+    locks_dir = repo_path / '.marco' / 'locks'
+    
+    with FileLock(locks_dir, "refs_update"):
+        refs = {}
+        if refs_file.exists():
+            refs = json.loads(refs_file.read_text(encoding='utf-8'))
+        for t in tags:
+            refs[t] = version_id
+        refs_file.write_text(json.dumps(refs, indent=2), encoding='utf-8')
+
+def list_tags(repo_path: Path) -> Dict[str, str]:
+    refs_file = Path(repo_path) / '.marco' / 'refs.json'
+    if refs_file.exists():
+        return json.loads(refs_file.read_text(encoding='utf-8'))
+    return {}
+
+def resolve_version(ref: str, repo_path: Path) -> str:
+    repo_path = Path(repo_path)
+    versions_dir = repo_path / '.marco' / 'versions'
+    
+    if (versions_dir / ref).exists():
+        return ref
+        
+    tags = list_tags(repo_path)
+    if ref in tags:
+        return tags[ref]
+        
+    matches = [d.name for d in versions_dir.iterdir() if d.is_dir() and d.name.startswith(ref)]
+    if len(matches) == 1:
+        return matches[0]
+    elif len(matches) > 1:
+        raise ValueError(f"Ambiguous short hash '{ref}'. Matches: {matches}")
+        
+    raise ValueError(f"Version '{ref}' not found.")
+
+def list_versions(repo_path: Path) -> List[dict]:
+    versions_dir = Path(repo_path) / '.marco' / 'versions'
+    if not versions_dir.exists():
+        return []
+    
+    tags = list_tags(repo_path)
+    tag_map = {}
+    for t, v in tags.items():
+        tag_map.setdefault(v, []).append(t)
+        
+    out = []
+    for d in versions_dir.iterdir():
+        if d.is_dir() and (d / 'manifest.json').exists():
+            man = json.loads((d / 'manifest.json').read_text(encoding='utf-8'))
+            man['tags'] = tag_map.get(d.name, [])
+            out.append(man)
+    out.sort(key=lambda x: x.get('created_at', ''), reverse=True)
+    return out
+
+def export_version(version_ref: str, dest_dir: Path, repo_path: Path):
+    vid = resolve_version(version_ref, repo_path)
+    version_dir = Path(repo_path) / '.marco' / 'versions' / vid
+    dest_dir = Path(dest_dir)
+    dest_dir.mkdir(parents=True, exist_ok=True)
+    
+    tar_path = dest_dir / f"marco_version_{vid[:8]}.tar.gz"
+    with tarfile.open(tar_path, "w:gz") as tar:
+        tar.add(version_dir, arcname=vid)
+    return tar_path
+
+def import_version(tar_path: Path, repo_path: Path):
+    tar_path = Path(tar_path)
+    repo_path = Path(repo_path)
+    versions_dir = repo_path / '.marco' / 'versions'
+    locks_dir = repo_path / '.marco' / 'locks'
+    
+    if not versions_dir.exists():
+        raise RuntimeError("Marco repo not initialized.")
+        
+    with FileLock(locks_dir, "import_version"):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            with tarfile.open(tar_path, "r:gz") as tar:
+                for member in tar.getmembers():
+                    if member.name.startswith('/') or '..' in member.name:
+                        raise ValueError("Tarball contains illegal paths.")
+                tar.extractall(path=tmpdir)
+                
+            extracted = list(Path(tmpdir).iterdir())
+            if len(extracted) != 1 or not extracted[0].is_dir() or len(extracted[0].name) != 64:
+                raise ValueError("Invalid marco tarball format.")
+                
+            vid = extracted[0].name
+            dest_dir = versions_dir / vid
+            if dest_dir.exists():
+                print(f"Version {vid} already exists.")
+                return vid
+                
+            shutil.copytree(extracted[0], dest_dir)
+            return vid

marco_dvcs-0.1.0/marco/web/app.py

@@ -0,0 +1,52 @@
+import argparse
+from pathlib import Path
+from flask import Flask, render_template, request, jsonify
+from marco.core import repository
+import json
+
+app = Flask(__name__)
+REPO_PATH = Path.cwd()
+
+@app.route('/')
+def index():
+    try:
+        versions = repository.list_versions(REPO_PATH)
+    except Exception:
+        versions = []
+    return render_template('index.html', versions=versions)
+
+@app.route('/version/<vid>')
+def version_detail(vid):
+    repo_path = REPO_PATH
+    try:
+        resolved_vid = repository.resolve_version(vid, repo_path)
+    except ValueError:
+        return "Version not found", 404
+        
+    v_dir = repo_path / '.marco' / 'versions' / resolved_vid
+    manifest = json.loads((v_dir / 'manifest.json').read_text(encoding='utf-8'))
+    metrics = json.loads((v_dir / 'metrics.json').read_text(encoding='utf-8'))
+    config = json.loads((v_dir / 'config.json').read_text(encoding='utf-8'))
+    
+    tags = repository.list_tags(repo_path)
+    v_tags = [t for t, v in tags.items() if v == resolved_vid]
+    
+    return render_template('version.html', manifest=manifest, metrics=metrics, config=config, tags=v_tags)
+
+@app.route('/api/versions')
+def api_versions():
+    return jsonify(repository.list_versions(REPO_PATH))
+
+def main():
+    parser = argparse.ArgumentParser(description="Marco Web Dashboard")
+    parser.add_argument("--port", type=int, default=5000)
+    parser.add_argument("--repo", type=str, default=".")
+    args = parser.parse_args()
+    
+    global REPO_PATH
+    REPO_PATH = Path(args.repo).resolve()
+    
+    app.run(port=args.port, debug=True)
+
+if __name__ == "__main__":
+    main()

marco_dvcs-0.1.0/marco_dvcs.egg-info/PKG-INFO

@@ -0,0 +1,123 @@
+Metadata-Version: 2.4
+Name: marco-dvcs
+Version: 0.1.0
+Summary: A minimal dataset versioning system for text data with a focus on reproducibility.
+Home-page: https://github.com/Team-Marco-ACM/marco-package
+Author: Your Name
+Author-email: your.email@example.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: Flask
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Marco Dataset Versioning System
+
+A minimal dataset versioning system for text data with a strong focus on reproducibility and transparency. Treat your text datasets like code — immutable, versioned, reproducible, and explainable.
+
+Marco acts as a lightweight Python library, meaning you can initialize it in *any* machine learning project folder to safely version and preprocess your datasets without altering your original files.
+
+## 🚀 Installation (Linux / MacOS)
+
+On modern Linux environments (like Arch Linux, Ubuntu 23.04+), Python packages must be installed in a Virtual Environment (PEP 668) to prevent conflicts with your system packages. 
+
+Follow these steps to safely install Marco into your ML project:
+
+1. **Clone this repository** to your local machine:
+   ```bash
+   git clone https://github.com/your-username/marco.git
+   cd marco
+   ```
+
+2. **Navigate to the ML project folder** where you want to train your model (e.g. your bag-of-words project):
+   ```bash
+   cd ~/projects/my-bag-of-words-model
+   ```
+
+3. **Create and activate a Python Virtual Environment**:
+   ```bash
+   # Create a virtual environment named 'venv'
+   python3 -m venv venv
+   
+   # Activate it (You must do this every time you open a new terminal in this folder)
+   source venv/bin/activate
+   ```
+   *(You should now see `(venv)` at the start of your terminal prompt!)*
+
+4. **Install Marco**:
+   ```bash
+   # Point pip to the directory where you cloned the marco repository
+   pip install -e /path/to/marco
+   ```
+
+---
+
+## 🛠️ Usage Guide
+
+Once `marco` is installed in your virtual environment, you have access to the full CLI!
+
+### 1. Initialize a Repository
+Initialize Marco tracking in your current directory. This creates a `.marco/` data versioning environment specific to that project.
+```bash
+marco init
+```
+
+### 2. Create an Immutable Version 
+Upload a text/CSV/TSV dataset to create an immutable version. Marco will compute a cryptographically secure SHA-256 hash using the raw data + the preprocessing configuration.
+
+**Interactive Mode:**
+If you don't supply a configuration file, Marco will interactively guide you through building the preprocessing pipeline (Lowercasing, Tokenization, Stopwords Removal, Deduplicating).
+```bash
+marco upload my_dataset.csv -t v1-raw
+```
+
+**Config Mode:**
+```bash
+marco upload my_dataset.csv -c my_config.json -t v1-processed
+```
+
+### 3. List Versions
+View all the versions you've created, along with their tags and timestamps.
+```bash
+marco list
+```
+
+### 4. Start the Visual Dashboard
+Open the Flask-powered web dashboard to view dataset metadata, differences between versions, and complete data lineage trees.
+```bash
+python -m marco.web.app --port 5000
+```
+Open `http://localhost:5000` in your web browser.
+
+### 5. Export / Import Versions
+Easily share dataset versions with teammates by packing them into `.tar.gz` files.
+```bash
+# Export version 'v1-raw' to the 'exports' folder
+marco export v1-raw ./exports/
+
+# Import an archive sent to you by a coworker
+marco import ./exports/marco_version_e5e0b767.tar.gz
+```
+
+---
+
+## 🧠 Architecture Overview
+
+Marco decouples logic from the file system. All core engine operations sit inside `marco/core/`, including:
+- `locker.py`: File-based concurrency control using `.lock` files.
+- `repository.py`: CRUD operations for dataset versions and `refs.json` tagging.
+- `preprocessor.py`: A robust Directed Acyclic Graph (DAG) preprocessing engine.
+
+Have fun building safer machine learning pipelines!

marco_dvcs-0.1.0/marco_dvcs.egg-info/SOURCES.txt

@@ -0,0 +1,22 @@
+README.md
+pyproject.toml
+setup.py
+marco/__init__.py
+marco/__main__.py
+marco/cli/__init__.py
+marco/cli/interactive.py
+marco/cli/main.py
+marco/cli/web_serve.py
+marco/core/__init__.py
+marco/core/locker.py
+marco/core/preprocessor.py
+marco/core/repository.py
+marco/ui/__init__.py
+marco/web/__init__.py
+marco/web/app.py
+marco_dvcs.egg-info/PKG-INFO
+marco_dvcs.egg-info/SOURCES.txt
+marco_dvcs.egg-info/dependency_links.txt
+marco_dvcs.egg-info/entry_points.txt
+marco_dvcs.egg-info/requires.txt
+marco_dvcs.egg-info/top_level.txt

marco_dvcs-0.1.0/marco_dvcs.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

marco_dvcs-0.1.0/marco_dvcs.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+marco = marco.cli.main:main

marco_dvcs-0.1.0/marco_dvcs.egg-info/requires.txt

@@ -0,0 +1,3 @@
+pandas
+numpy
+Flask

marco_dvcs-0.1.0/marco_dvcs.egg-info/top_level.txt

@@ -0,0 +1 @@
+marco

marco_dvcs-0.1.0/pyproject.toml

@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=42", "wheel"]
+build-backend = "setuptools.build_meta"

marco_dvcs-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

marco_dvcs-0.1.0/setup.py

@@ -0,0 +1,35 @@
+import os
+from setuptools import setup, find_packages
+
+# Read the contents of your README file
+this_directory = os.path.abspath(os.path.dirname(__file__))
+with open(os.path.join(this_directory, "README.md"), encoding="utf-8") as f:
+    long_description = f.read()
+
+setup(
+    name="marco-dvcs",  # Changed to 'marco-dvcs' as 'marco' might be taken on PyPI
+    version="0.1.0",
+    author="Your Name",
+    author_email="your.email@example.com",
+    description="A minimal dataset versioning system for text data with a focus on reproducibility.",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/Team-Marco-ACM/marco-package",
+    packages=find_packages(),
+    install_requires=[
+        "pandas",
+        "numpy",
+        "Flask",
+    ],
+    entry_points={
+        "console_scripts": [
+            "marco=marco.cli.main:main",
+        ],
+    },
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+    ],
+    python_requires=">=3.8",
+)