@dforge-core/dforge-mcp 0.2.45 → 0.2.47

package/skills/dforge-1c-import/references/measurements.md

@@ -0,0 +1,210 @@
+# The measurements
+
+Each decision in SKILL.md has a query that settles it. They are written against a loaded
+dForge tenant (Postgres) because that is where both sides are comparable — the 1C rows and
+what the platform writes. Where the question can only be asked of the infobase, the query
+is against the export (DuckDB over `out/data/`, via `store.connect()`) and says so.
+
+Adapt names; the shapes are the point.
+
+## Contents
+
+1. Does this register need a collapse?
+2. Which kind is it?
+3. Would accumulation ever accumulate?
+4. Did the collapse actually happen?
+5. Is a guard in the handler alive?
+6. Does anything read this?
+7. Is a column's type a lie?
+8. Is a text column really full?
+9. Which costing method does the base use?
+
+---
+
+## 1. Does this register need a collapse?
+
+The handler's `Свернуть` is the signal; these two confirm it and size it.
+
+```sql
+-- how often the lines of one document repeat a dimension
+select repeats, count(*) as documents from (
+  select <doc_fk>, count(*) - count(distinct <dim>) as repeats
+  from <line_table> group by 1) x
+group by 1 order by 1 limit 10;
+
+-- lines in, rows 1C actually wrote out, for the same documents
+select sum(lines) as lines, sum(rows_1c) as rows_1c from (
+  select d.<pk> id,
+    (select count(*) from <line_table> l where l.<doc_fk> = d.<pk>) as lines,
+    (select count(*) from <register> r where r.recorder = d.<pk>) as rows_1c
+  from <documents> d) y;
+```
+
+`lines > rows_1c` means the collapse is load-bearing.
+
+## 2. Which kind is it?
+
+```sql
+select count(*) as rows,
+       count(distinct (<dims>)) as distinct_keys,
+       count(distinct recorder) as writing_documents
+from <register>;
+```
+
+`rows = distinct_keys` means one row per key — a table of records, not a journal. Combine
+with question 1: one row per key *and* repeating lines means the writer collapses, which
+is `balance` in dForge regardless of the 1C family.
+
+## 3. Would accumulation ever accumulate?
+
+```sql
+select docs_per_key, count(*) as keys from (
+  select <key columns>, count(*) as docs_per_key
+  from <documents>
+  where coalesce(deletion_mark, false) = false
+  group by <key columns>) z
+group by 1 order by 1;
+```
+
+One output row showing `1` retires the objection that `balance` would add two openings
+together. Record the number in the declaration; it is the answer to the question the next
+reader will ask.
+
+## 4. Did the collapse actually happen?
+
+Two traps here, and they pull in opposite directions.
+
+Resource **totals** match whether the engine collapsed or not, because a sum is a sum. So
+totals alone cannot show a collapse. But row counts alone cannot either: against a
+register that already holds the migrated rows, a posting that did **nothing** adds no rows
+and creates no duplicate keys — and so passes a check built only on counts. A posting that
+updated the *wrong* existing key passes it too.
+
+What settles it is the expected delta **per full dimension tuple**, checked in both
+directions. Compute what the lines say each tuple should move, before posting:
+
+```sql
+-- expected, read through the binding's own map and headerMap
+select <dims>, sum(<resource>) as expected
+from <line_table> l join <documents> d on d.<pk> = l.<doc_fk>
+where d.<pk> = '<doc>'
+group by <dims>;
+```
+
+Then post, and compare per tuple rather than in aggregate:
+
+```sql
+select r.<dims>, r.<resource> - b.<resource> as moved
+from <register> r join <baseline snapshot> b using (<dims>);
+```
+
+Three things must hold together, and any one alone is weak:
+
+1. every expected tuple moved by exactly its expected amount;
+2. no tuple outside the expected set moved at all;
+3. after unposting, every tuple is back at the baseline.
+
+Point 2 is what catches an update to the wrong key; point 3 is what catches a posting that
+silently did nothing, because a no-op also unposts to nothing and the baseline never
+moved. Snapshot the register's rows for the affected tuples before posting so the
+comparison has something to be against.
+
+And separately, since it is cheap:
+
+```sql
+select count(*) as keys_with_two_rows from (
+  select 1 from <register> group by <dims> having count(*) > 1) x;
+```
+
+## 5. Is a guard in the handler alive?
+
+`Если Стр.Количество <> 0` guards a loop. Has it ever fired?
+
+```sql
+select count(*) as lines,
+       count(*) filter (where <col> = 0)  as would_be_skipped,
+       count(*) filter (where <col> <> 0) as actually_written
+from <line_table>;
+```
+
+Zero skipped over years of data means the guard is dead code. Porting it as a platform
+feature would be paying for nothing.
+
+## 6. Does anything read this?
+
+A grep over the extracted configuration modules, not a query. Count the objects naming the
+register, split by whether the object is the client's own (lowercase prefix before a
+capital) or the vendor's, and look at what appears near each mention. If the only mentions
+are the writing document and `УдалитьДвиженияРегистратора`, nothing reads it.
+
+Do this before carrying a journal, a recorder link, or any mechanism that exists to
+support a reader.
+
+## 7. Is a column's type a lie?
+
+Sweep every text column at once rather than meeting them one at a time. Build the union
+from `information_schema.columns` in the module schemas, then:
+
+```sql
+select '<schema>.<table>.<col>' as col, count(*) as n
+from "<schema>"."<table>"
+where "<col>" ~ '^[0-9]{2}\.[0-9]{2}\.[0-9]{4}( |$)'
+```
+
+A hit is a date the 1C author declared as a string. Expect very few — in the base measured,
+one column out of 611 — which is what makes the sweep worth running once and never
+worrying again.
+
+## 8. Is a text column really full?
+
+```sql
+select count(*) as rows,
+       count(*) filter (where <col> is null)  as nulls,
+       count(*) filter (where <col> = '')     as empty_strings,
+       count(*) filter (where coalesce(<col>, '') <> '') as actually_filled
+from <table>;
+```
+
+`nulls = 0` while `actually_filled = 0` is the signature of a column read from a field the
+infobase never writes. It is invisible to `count(column)`.
+
+## 9. Which costing method does the base use?
+
+The accounting policy register will name an enum position and the configuration text may
+not resolve it. The movements answer it several ways, and agreement across them is the
+point — any one alone can mislead.
+
+```sql
+-- A. batches touched by one write-off. Average costing writes off proportionally across
+--    every batch in stock, so a heavy concentration at 1 rules it out.
+select batches, count(*) as expenses from (
+  select _RecorderRRef, <item>, <warehouse>, count(distinct <batch>) as batches
+  from <batch_register>
+  where _RecordKind = 1 and _Period >= ? and _Period < ?
+  group by 1,2,3) x
+group by 1 order by 1;
+
+-- B. distinct unit costs per item and warehouse within one month.
+--    A period-weighted average gives exactly one.
+select prices, count(*) from (
+  select <item>, <warehouse>,
+         count(distinct round(<amount> / nullif(<qty>, 0), 4)) as prices
+  from <batch_register>
+  where _RecordKind = 1 and _Period >= ? and _Period < ? and <qty> <> 0
+  group by 1,2) y
+group by 1 order by 1;
+
+-- C. was there a choice at all? (validity check: zero-stock cases mean the
+--    balance arithmetic is wrong and nothing below can be believed)
+--    then, among those with a choice, was the oldest batch in stock taken?
+--    FIFO takes the oldest; LIFO the newest; neither dominating means no rule
+--    is followed and the batch is effectively identified rather than queued.
+```
+
+Read C carefully: "the oldest batch ever received" is a different question from "the oldest
+batch that still had stock at that moment", and only the second is FIFO. The first gives a
+muddle and will mislead you.
+
+Whatever the answer, it changes the question put to the client rather than replacing it:
+not *"which method do you use"* — the base answers that — but *"is that what you meant, and
+should it stay that way"*. A rule nobody enforces may be accepted practice or an accident.

package/skills/dforge-1c-import/references/print-forms.md

@@ -0,0 +1,148 @@
+# Porting 1C print forms
+
+The platform's side of this (manifest, context, snippets, Scriban versus LiquidJS) is in
+`dforge://reference/print-templates`; queries are in `dforge://reference/queries`. This
+file covers what a 1C form brings with it.
+
+## What a 1C form is
+
+1C prints in two steps:
+
+1. The document's print procedure (`Печать…`, often in a shared module) computes almost
+   every value. It folds lines by product and series, takes places and packs through the
+   series' packaging, looks prices up at the date, totals, and assembles the
+   organization's requisites into one text.
+2. A spreadsheet template (макет, `MOXCEL`) is filled area by area with those values.
+
+So the template alone is not the form. **Read the procedure first.** It holds the
+requirement. The template is only the layout.
+
+The dForge form mirrors the two steps:
+
+| 1C | dForge |
+|---|---|
+| the print procedure | the template's datasets: a query, with F columns for per-line arithmetic |
+| the template (макет) | an HTML template with its CSS |
+| the shared printing library (requisites, contact info, amounts in words) | shared datasets and snippets, e.g. the platform's `common-words` for amounts and quantities in words |
+
+## Reading the template
+
+A template is the object's metadata group `{3daea016-69b7-4ed4-9453-127911372fe6, n,
+guid…}`. Each template's `<guid>.txt` names it and its body is `<guid>.0.txt`. A spreadsheet
+body starts `MOXCEL` and continues as a brace list:
+
+- **Rows** with their cells. A cell is one of:
+  - `{0, format}` — empty;
+  - `{16, format, {1,1,{lang,text}}}` — text, or a parameter when `lang` is `""`;
+  - `{20, …, {"U"}, …}` — a template string whose `[Name]` parts are parameters.
+- **Column sets**, **merges** and **named areas**. An area is rows (kind 1) or columns
+  (kind 2), with inclusive bounds.
+- **Formats.** Each is a bit mask followed by its values, low bit first: font index (1),
+  left/top/right/bottom border (2/4/8/16), height (64), width (128). The remaining bits
+  (alignment, placement) can be carried by number.
+- **Fonts.**
+
+Dump each template to HTML and JSON: areas, cells, parameters, merges, widths, fonts,
+borders. **Author the form from the dump; do not convert it.** 1C's layout is a grid of up
+to ~50 narrow columns. A faithful HTML copy of it is unmaintainable. Keep the form's
+logical columns and labels.
+
+Which areas the procedure outputs, and in what order, is the form's real structure. Typical
+areas are header, line, subtotal per group, total and signatures.
+
+## Inventory before porting
+
+- **List the forms per document from the print command's handler**, not from the
+  templates. An object can carry dead templates. Templates the code names only for dates
+  before some year print only old documents: record them and skip them.
+- **Count how often each variant fires** on recent documents. A variant is a template
+  chosen by a flag on the buyer or the contract, or a "per code" text. A variant used by
+  278 documents since last year is worth a template. One used by none may still be worth a
+  column.
+- **External print processings** (`ВнешниеОбработки`) live in the infobase as stored
+  binaries, not in the configuration. Inventory them separately.
+
+## Reproduce what 1C prints, including its slips
+
+The client compares the new printout to the old one. Where 1C's code has a defect, the
+printout has the defect, and the client knows the printout. Reproduce it, and say so in
+a comment at the place. Common ones:
+
+- **A misspelt variable** makes a lookup find nothing. For example, a phone kind read from
+  a misnamed variable means no phone ever prints. Print nothing.
+- **A field list with spaces** (`"НомерСчета, Банк, АдресБанка"`) passed where 1C expects
+  no spaces fills only the first field.
+- **A test on the wrong attribute**, such as an IBAN check on the account's name instead
+  of its number. Do the same test.
+- **Mixed Cyrillic and Latin letters** in a literal (`с` against `c`). Copy them byte for
+  byte.
+- **Language codes** in `НСтр("ru = '…'; en = '…'", Lang)` are matched case-insensitively,
+  so `"Ru"` finds `ru`.
+
+If the client later asks to fix one, it is a one-line change with the comment pointing
+at it.
+
+## Formatting numbers and dates as 1C does
+
+| 1C | output | Postgres equivalent |
+|---|---|---|
+| `Строка(n)`, `Формат(n)` without a format | thousands grouped by nbsp, decimal comma, no trailing zeros | `replace(replace(rtrim(rtrim(to_char(n,'FM999G999G999G990D9999999999'),'0'),'.'),',',E' '),'.',',')` (the `G`/`D` result depends on `lc_numeric`; check it) |
+| `Формат(n, "ЧДЦ=2")` | always 2 decimals | `to_char(n, 'FM999G999G990D00')` then the same separators |
+| `Формат(n, "ЧРД=.")` | decimal point | the replacement above with `.` |
+| `Формат(n, "ЧГ=0")` | no grouping | drop `G` |
+| `Формат(d, "ДФ=dd.MM.yyyy")` | `13.01.2016` | `to_char(d, 'DD.MM.YYYY')` |
+| a date parameter output as is | `13.01.2016 9:05:00` (hour without a leading zero) | `to_char(d, 'DD.MM.YYYY FMHH24:MI:SS')` |
+
+Before choosing the precision a price prints with, measure how many stored values actually
+have more decimals. Do not trust the declared precision: stored prices can carry a third
+decimal on most rows.
+
+Amounts and quantities in words should come from a shared snippet (the platform's
+`common-words`), not per-form code. Verify it on the client's real values against 1C's
+`ЧислоПрописью` output: thousands of values, every case. Expect differences only where 1C
+is ungrammatical, and agree them with the client.
+
+## Datasets: queries, and where a query falls short
+
+Prefer a query dataset. It runs under the user's rights and folder filters, which a stored
+procedure bypasses.
+
+- **Put per-line arithmetic into F columns on the line entity**: weight = places × place
+  weight, packs, the VAT percentage through a `SWITCH`, a group section through `IN`.
+- **Let the query group and sum** (`agg: group | sum`).
+- **Accumulate totals in the template.** The server engine has no `where`, `sum` or
+  `group_by` filters, but `plus`, `times`, `round` and `ceil` work.
+- **A dataset is always an array.** Take a header row with
+  `{% assign h = header | first %}`.
+
+A query cannot (yet) express these, so the dataset stays a procedure until it can:
+
+- a value in force at a date (`СрезПоследних`: prices, positions, codes, shelf lives);
+- a filter through a reverse set or a polymorphic reference;
+- string aggregation;
+- a join on columns that are not a reference;
+- formulas over aggregates.
+
+1C rounds after folding (`round(price × SUM(qty))`). A query sums a per-line formula. The two
+differ only where lines fold. Measure how many documents that touches, then agree it.
+
+## Verifying a ported form
+
+1. **Compare the dataset with the procedure's output.** Pick tens of random recent
+   documents and compare every field, line by line. Zero differences is the bar. An F
+   column rewritten from a procedure gets the same check.
+2. **Render previews on real documents** for every variant: each language, each flag,
+   with and without the optional blocks. Screenshot them against 1C's printout where one
+   is available.
+3. **Measure the data each block depends on.** How many recent documents have the date,
+   the indicators, the shelf life? A block empty on every document is either 1C's state
+   (record it) or a column read from the wrong source (fix it).
+
+## Generating templates
+
+When several forms share blocks (requisites, a lines table, signatures, a certificate
+body), write the templates from one generator. Do not hand-copy HTML. Keep the generator
+in the project, regenerate on every change, and never edit its output.
+
+In Python, keep each formatted literal its own expression. Adjacent string literals
+concatenate **before** `%` applies, which produces format errors far from the cause.

package/skills/dforge-1c-import/scripts/1c.py

@@ -0,0 +1,178 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+One way in, so the split between library and project stays a fact rather than a habit.
+
+    python tools/1c.py <command> [args...]
+    python tools/1c.py --list          # what there is
+    python tools/1c.py --boundary      # does the library still stand on its own?
+
+`tools/lib` is 1C and dForge: how a configuration is stored, how its metadata parses,
+what the physical tables mean, and how rows are copied and checked. It is meant to be
+lifted out and pointed at a different infobase.
+
+`tools/project` is this infobase: which object belongs to which module, which 1C column
+feeds each ported one, which regulated statements it holds, this company's own proper
+nouns, and the modules written by hand. `DF1C_PROJECT` points the library at a different
+one; `binding.py` lists the whole of what a project has to provide.
+
+Both directories go on the path, so the modules import each other by plain name and a
+move between them costs no edits. That convenience is also the hazard, so `--boundary`
+checks two things rather than trusting either: that no library module IMPORTS a project
+one, and that no library module NAMES this project's world - its server, its infobase,
+its tenant, its company's abbreviations. A literal is the quiet way to bind a library to
+one configuration, and it is how `DF1C_DB` came to default to `utp_biscuit` for months.
+
+Because both directories are on one path, a project module may not take a name the
+library already uses: `regulated.py` rather than `forms.py`, `vocabulary.py` rather than
+`terms.py`. `--boundary` catches the shadowing as an import crossing.
+"""
+from __future__ import print_function
+
+import io
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+LIB = os.path.join(HERE, "lib")
+PROJECT = os.path.abspath(os.environ.get("DF1C_PROJECT") or os.path.join(HERE, "project"))
+
+for entry in (PROJECT, LIB):
+	if entry not in sys.path:
+		sys.path.insert(0, entry)
+
+
+def commands():
+	"""Every runnable module, library or project, mapped to (path, first docstring line)."""
+	found = {}
+	for directory in (LIB, PROJECT):
+		for name in sorted(os.listdir(directory)):
+			if not name.endswith(".py") or name.startswith("_") or name[:-3] in found:
+				continue
+			path = os.path.join(directory, name)
+			summary = _runnable(path)
+			if summary is not None:
+				found[name[:-3]] = (path, summary)
+	return found
+
+
+def _runnable(path):
+	"""The module's first docstring line if it can be run as a script, else None."""
+	with io.open(path, encoding="utf-8") as f:
+		text = f.read()
+	# Anything runnable: most define main(), a few only a __main__ block.
+	if "def main(" not in text and "__name__" not in text:
+		return None
+	if '"""' not in text:
+		return ""
+	body = text.split('"""', 2)[1].strip().splitlines()
+	return (body[0] if body else "").strip()
+
+
+def boundary():
+	"""Two ways the library can stop standing on its own, both read off the source.
+
+	IMPORTS. The path puts both directories in reach, so nothing stops a library module
+	importing `contours` directly and nothing would complain until someone tried to lift
+	the library out.
+
+	NAMES. An import is the loud way to bind to one infobase; a literal is the quiet one.
+	`DF1C_DB` defaulting to `utp_biscuit` and `DF_TENANT` to `hbf1` sat in the library for
+	a long while, harmless with one infobase and a trap with two - point the tools at a
+	second configuration, forget one variable, and they read the first one with
+	right-looking output. So the names of THIS project's world are refused in the library,
+	including in its comments, since a stale comment is how the next reader learns the
+	wrong thing.
+
+	The list is read from the project rather than written here, so a project that renames
+	itself does not have to remember this file.
+	"""
+	import ast
+	import re
+
+	project_modules = set(n[:-3] for n in os.listdir(PROJECT) if n.endswith(".py"))
+
+	# What this project calls itself: the infobase, the tenant and its own proper nouns.
+	reserved = set()
+	for variable in ("DF1C_DB", "DF1C_SERVER", "DF_TENANT"):
+		value = os.environ.get(variable)
+		if value:
+			reserved.add(value.lower())
+	env_example = os.path.join(os.path.dirname(os.path.abspath(PROJECT)), os.pardir,
+	                           ".env.example")
+	if os.path.exists(env_example):
+		with io.open(env_example, encoding="utf-8") as f:
+			for line in f:
+				match = re.match(r"\s*(DF1C_DB|DF1C_SERVER|DF_TENANT)\s*=\s*(\S+)", line)
+				if match:
+					reserved.add(match.group(2).lower())
+	terms = os.path.join(PROJECT, "terms.py")
+	if os.path.exists(terms):
+		with io.open(terms, encoding="utf-8") as f:
+			for value in re.findall(r'u"([^"]+)"\s*:', f.read()):
+				if len(value) > 3:      # two- and three-letter forms collide with real words
+					reserved.add(value.lower())
+
+	problems = []
+	for name in sorted(os.listdir(LIB)):
+		if not name.endswith(".py"):
+			continue
+		with io.open(os.path.join(LIB, name), encoding="utf-8") as f:
+			text = f.read()
+		tree = ast.parse(text)
+		for node in ast.walk(tree):
+			named = []
+			if isinstance(node, ast.Import):
+				named = [a.name.split(".")[0] for a in node.names]
+			elif isinstance(node, ast.ImportFrom) and node.module:
+				named = [node.module.split(".")[0]]
+			for target in named:
+				if target in project_modules:
+					problems.append("lib/%s imports %s - go through binding.py instead"
+					                % (name, target))
+		# paths.py explains why these names are refused, so it is allowed to name them.
+		if name == "paths.py":
+			continue
+		lowered = text.lower()
+		for word in sorted(reserved):
+			if word in lowered:
+				problems.append("lib/%s names %r - that is the project's, not the library's"
+				                % (name, word))
+	return problems
+
+
+def main():
+	args = sys.argv[1:]
+	if not args or args[0] in ("-h", "--help", "--list"):
+		import paths
+		paths.show()
+		print()
+		for name, (_, summary) in sorted(commands().items()):
+			print("  %-16s %s" % (name, summary[:60]))
+		return 0
+
+	if args[0] == "--boundary":
+		problems = boundary()
+		for line in problems:
+			print(line)
+		print("%d crossing(s) of the library/project boundary" % len(problems))
+		return 1 if problems else 0
+
+	name, rest = args[0], args[1:]
+	found = commands()
+	if name not in found:
+		print("no such command: %s (try --list)" % name)
+		return 2
+	# run_path rather than calling main(): a few tools only have a __main__
+	# block, and this runs every one of them exactly as a script would.
+	import runpy
+	sys.argv = [name] + rest
+	try:
+		runpy.run_path(found[name][0], run_name="__main__")
+	except SystemExit as stop:
+		return stop.code or 0
+	return 0
+
+
+if __name__ == "__main__":
+	sys.exit(main() or 0)