@dforge-core/dforge-mcp 0.2.45 → 0.2.47

package/skills/dforge-1c-import/scripts/README.md

@@ -0,0 +1,164 @@
+# The 1C porting toolkit
+
+A Python library that reads a 1C:Enterprise 8 infobase, either a server infobase on MS SQL
+Server or a file infobase (`1Cv8.1CD`). It needs no 1C software and no licence. From the
+infobase it:
+
+- generates dForge modules;
+- copies the rows;
+- checks the result.
+
+It was built on a real port: an УТП-based configuration, 16 contours, 30 M+ rows,
+reconciled to the row and the kopeck. It is meant to be pointed at the next infobase
+unchanged.
+
+```
+scripts/
+  1c.py                 the one entry point: python 1c.py <command> [args]
+  lib/                  the library: 1C storage, metadata, generation, loading, checks
+  project-template/     what a new port starts its project package from
+```
+
+## Two layers: export, then everything else
+
+Only one step reads 1C itself. `dump1c` exports the infobase to `out/`, and every later
+step reads that export, never the infobase.
+
+| layer | modules | reads | writes |
+|---|---|---|---|
+| export | `dump1c` · `source_mssql` · `source_file` · `onecd` | the infobase: MS SQL (`DF1C_SERVER`/`DF1C_DB`) or a `.1CD` file (`DF1C_FILE`) | `out/config/`, `dbnames.json`, `schema.json`, `data/<table>.parquet`, `export.json` |
+| everything else | `store` and every other module | `out/`. Rows are queried with DuckDB through `store.connect()` | modules, the tenant, reports |
+
+The export looks the same whichever kind of infobase it came from:
+
+- names are spelt the MS SQL way (`_Reference12`, `_Fld345RRef`). A file infobase keeps
+  them in capitals, and DBNames supplies the spelling back;
+- the schema uses MS SQL type names (`binary(16)`, `nvarchar`, `numeric`, `datetime`);
+- booleans are `binary(1)`.
+
+Rows are kept as 1C stores them. `export.json` records the infobase's year offset: usually
+2000 on MS SQL, always 0 in a file infobase. It also records the horizon (`MIN_ACTIVE_ROWVERSION()`)
+taken before each table was read.
+
+- **SQL over the export is DuckDB's.** That includes a project's `sources.QUERIES`. Quote
+  names with double quotes (case does not matter), use `LIMIT` instead of `TOP`, write
+  a binary constant as `from_hex('01')` instead of `0x01`, and use `CAST` instead of
+  `CONVERT`. A port whose queries were written in T-SQL has to rewrite them.
+- **`dump1c data --changes` reads only what 1C changed** since the export (or `--since
+  <day>`). It works table by table:
+  - objects are re-read by `_Version` above the mark, and deletions are found by key;
+  - tabular parts are re-read through their changed owners;
+  - register movements are re-read through their changed recorders;
+  - anything else is re-read whole.
+
+  It rewrites each table's file to what a full export would hold. On an infobase of 2.7 M
+  objects, it reads the keys in a few seconds instead of a full export of 1.5 h.
+
+  On an accounting register it also compares each recorder's count of inactive entries.
+  Switching entries on or off ("Включить/выключить проводки") rewrites them without
+  writing the document, so `_Version` does not show it.
+- **Totals and document journals are left out by default** (`_AccumRgT`, `_AccumRgTn`,
+  `_AccRgAT`, `_AccRgCT`, `_DocumentJournal`). 1C derives them from other tables, they
+  have no `_Version`, and `--changes` would read them in full on every run: 53.7 M of
+  313 M rows. Name one, or pass `--derived`, to export it. `seedstate verify` says which
+  totals table it needs.
+- **`delta` needs an MS SQL export.** In a file infobase `_Version` is a counter per table,
+  not a rowversion for the whole database, so a file export has no horizon. Re-export it
+  and reload with `migrate run <module> --truncate`.
+- **Read a file infobase while 1C is not writing to it.** Copy the `.1CD` first, or close
+  the infobase.
+
+## Library and project
+
+- **`lib/` knows 1C and dForge**: how a configuration is stored, how its metadata
+  parses, what the physical tables mean, and how rows are copied and checked.
+- **The project knows this infobase**:
+  - which object goes to which module (`contours.py`);
+  - which 1C column feeds an authored column (`sources.py`);
+  - the integrator's prefixes;
+  - the hand-written modules (`authored/<module>/`).
+
+`lib/binding.py` lists everything a project may provide. Only `contours` and `sources`
+are required.
+
+**Never put project facts in `lib/`.** That covers infobase names, tenant names, the
+company's abbreviations and prefixes. `python 1c.py --boundary` checks that no library
+module imports a project module or names the project's world.
+
+**Never hand-edit generated output.** Put the change in the project, either as a spec
+override or as an authored file under `authored/<module>/` that `genmodule` merges.
+Then regenerate. If a port keeps needing hand edits, the generator is wrong.
+
+## Setup
+
+- Python 3.7+.
+- `pyarrow` and `duckdb`, to write and read the export (on 3.7: `pyarrow==12.0.1`,
+  `duckdb==1.1.3`).
+- `pyodbc` and an ODBC driver for SQL Server (driver 17 or 18), only to export a server
+  infobase. A file infobase needs nothing more.
+- `psycopg2`, to load and check the tenant.
+- `jsonschema`, for `validate`.
+- Node with `liquidjs` in `PRINT_NODE_DIR`, only for `printgen preview`.
+
+On Windows, set `PYTHONIOENCODING=utf-8`.
+
+Lay out a port directory, copy `project-template/.env.example` to `<root>/.env` and fill it in, and export `DF1C_ROOT` / `DF1C_PROJECT` in the shell:
+
+```
+myport/
+  .env                  credentials (never committed)
+  project/              copied from project-template/, then filled in
+  out/                  what the extraction writes (DF1C_OUT, default <root>/out)
+  modules/              generated modules (DF1C_MODULES, default <root>/modules)
+```
+
+```
+DF1C_ROOT=/path/to/myport
+DF1C_PROJECT=/path/to/myport/project
+DF1C_SERVER=<sql server>   DF1C_DB=<infobase db>   DF1C_USER= DF1C_PASS=   # read-only login; Windows auth when USER is empty
+DF1C_FILE=<path to 1Cv8.1CD, or its folder>                                 # a file infobase instead of the two above
+DF_TENANT=<tenant code>    DF_PG_HOST= DF_PG_PORT= DF_PG_USER= DF_PG_PASS=  # the tenant's Postgres, for load and checks
+```
+
+None of these has a default. A default that named one infobase would make the toolkit
+quietly read or write that infobase when a variable was forgotten.
+
+## The pipeline
+
+Each step reads what the previous one wrote to `out/`. Every command has `--help` or a
+docstring with its options, and `python 1c.py --list` lists them all.
+
+| # | step | command |
+|---|---|---|
+| 1 | Export the infobase: configuration, DBNames, schema, rows | `dump1c probe` · `dump1c all`, or one at a time: `dump1c dbnames` · `schema` · `extract` · `data [table ...]` |
+| 2 | Parse the metadata, resolve type GUIDs, join the physical schema | `meta allobjects` · `typemap` · `analyze` · `attrs <Object>` · `explore <Object>` |
+| 3 | Extract the module texts (posting handlers, print procedures) | `extract_code` · `unpack <Object> --code` |
+| 4 | Inventory: what exists, what is used, what is the client's | `inventory` · `docusage` · `custom` · `sizing` · `subsystems` |
+| 5 | Read the posting handlers: which registers each document writes and from where | `handlers [--client]` · `callgraph` · `recorders` · `recorded` · `idioms` · `spec` |
+| 6 | Decide the module boundaries | `subsystems --contours --acyclic`, copied into `project/contours.py` and corrected |
+| 7 | Generate and validate the modules | `autospec [module]` · `genmodule` · `validate` · `deps` · `layers` |
+| 8 | Install them | `npx -y @dforge-core/dforge-cli@latest module install ...`, or `rebuild --code <tenant> --install` |
+| 9 | Copy the rows and reconcile | `migrate plan\|check\|run\|verify\|fill <module>` · `integrity` · `columnmap <module>` |
+| 10 | Post state and registers | `posted [--mark]` · `posting` · `seedstate` · `totals` · `coverage` |
+| 11 | Catch up with what 1C changed (MS SQL exports only) | once: `dump1c data --objects` (delta needs every object's keys); then `dump1c data --changes`, `delta --baseline` · `delta [--apply]` |
+| 12 | Print forms and reports | `mxl list [--client]` · `mxl dump <Object>` · `printgen <spec>` · `printgen preview <spec> <form> <key>` · `reportgen <spec>` |
+| 13 | External reports and processings stored in the infobase | `external` |
+
+`rebuild --code <tenant>` runs steps 8–10 from nothing, which makes a tenant
+reproducible.
+
+## Rules the toolkit assumes
+
+- **The infobase is read-only.** Use a SELECT-only login. Read a file infobase from a copy.
+- **Open ad-hoc database connections in autocommit.** A read transaction left open
+  blocks the next install's `ALTER`.
+- **Resolve a physical column (`_FldNNN`) only through DBNames** (`attrs`, `columnmap`).
+  Never resolve it by cardinality.
+- **Stop on a 401 from the tenant.** Ask for a fresh login, then re-run
+  `posted --mark` for documents loaded in the meantime.
+
+## Source
+
+This copy is kept in step with the port it was developed on (`tools/lib`, `tools/1c.py`
+there). Fixes made in a port go back to the library here. A project's own files never
+come here.

package/skills/dforge-1c-import/scripts/lib/analyze.py

@@ -0,0 +1,83 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+Join the physical schema against DBNames to see which parts of the configuration
+the client actually uses. A stock UTP ships hundreds of catalogs and documents;
+only the populated ones are worth porting.
+"""
+import codecs
+import collections
+import json
+import os
+import re
+import sys
+
+import paths
+
+OUT = paths.OUT
+
+# Physical table families that carry business data, in the order we care about.
+DATA_KINDS = ("Reference", "Document", "InfoRg", "AccumRg", "AccRg", "Chrc", "Acc",
+              "Enum", "DocumentJournal", "Node", "BPr", "Task", "Const")
+
+SPLIT = re.compile(r"^_([A-Za-z]+?)(\d+)(?:_(.+))?$")
+
+
+def load(name):
+	with codecs.open(os.path.join(OUT, name), "r", "utf-8") as f:
+		return json.load(f)
+
+
+def classify(table_name):
+	"""_Document123_VT456 -> ('Document', 123, 'VT456')"""
+	m = SPLIT.match(table_name)
+	if not m:
+		return None, None, None
+	return m.group(1), int(m.group(2)), m.group(3)
+
+
+def main():
+	schema = load("schema.json")
+	tables = {t["name"]: t for t in schema["tables"]}
+
+	summary = collections.OrderedDict((k, {"tables": 0, "used": 0, "rows": 0}) for k in DATA_KINDS)
+	detail = collections.defaultdict(list)
+
+	for name, t in tables.items():
+		kind, number, suffix = classify(name)
+		if kind not in summary or suffix:          # skip tabular sections, roll them up separately
+			continue
+		rows = t["rows"] or 0
+		# a document's tabular sections belong to the document's footprint
+		child_rows = sum((tables[n]["rows"] or 0) for n in tables
+		                 if n.startswith("_%s%d_" % (kind, number)))
+		summary[kind]["tables"] += 1
+		summary[kind]["rows"] += rows
+		if rows:
+			summary[kind]["used"] += 1
+		detail[kind].append({"table": name, "number": number, "rows": rows,
+		                     "childRows": child_rows, "columns": len(t["columns"])})
+
+	print("%-18s %8s %8s %14s" % ("kind", "tables", "used", "rows"))
+	print("-" * 52)
+	for kind, s in summary.items():
+		if s["tables"]:
+			print("%-18s %8d %8d %14s" % (kind, s["tables"], s["used"], "{:,}".format(s["rows"])))
+
+	top = int(sys.argv[1]) if len(sys.argv) > 1 else 25
+	for kind in ("Document", "Reference", "AccumRg", "InfoRg"):
+		rows = sorted(detail[kind], key=lambda d: -(d["rows"] + d["childRows"]))[:top]
+		print("\n== %s: top %d by volume ==" % (kind, top))
+		for d in rows:
+			if not (d["rows"] or d["childRows"]):
+				continue
+			print("   %-22s rows=%-12s child=%-12s cols=%d"
+			      % (d["table"], "{:,}".format(d["rows"]),
+			         "{:,}".format(d["childRows"]), d["columns"]))
+
+	with codecs.open(os.path.join(OUT, "usage.json"), "w", "utf-8") as f:
+		json.dump({"summary": summary, "detail": detail}, f, ensure_ascii=False, indent="\t")
+
+
+if __name__ == "__main__":
+	main()

package/skills/dforge-1c-import/scripts/lib/attrs.py

@@ -0,0 +1,278 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+Extract the field list of a metadata object, with references resolved.
+
+Every typed thing in a 1C object file - a catalog attribute, a register dimension
+or resource, a tabular-section column - is encoded the same way:
+
+    ["2", ["1", ["0","0",<own guid>], "<Name>", ["1","ru","<Synonym>"], ...],
+          ["Pattern", <type...>]]
+
+so the collector looks for that shape anywhere in the tree rather than hard-coding
+per-kind offsets, and takes the nearest named ancestor as the tabular section.
+
+    python attrs.py <ObjectName>              one object, to the console
+    python attrs.py --model <file> <names...> several objects, to a markdown file
+"""
+from __future__ import print_function
+
+import codecs
+import json
+import os
+import paths
+import sys
+
+import meta
+
+OUT = meta.OUT
+
+PRIMITIVE = {"S": "string", "N": "number", "B": "bool", "D": "date"}
+
+# Collection class GUIDs -> what the fields in them mean, confirmed by checking a
+# register whose meaning is unambiguous (ТоварыНаСкладах: Количество is the resource,
+# Склад/Номенклатура the dimensions). Unknown GUIDs print truncated so they show up.
+ROLE = {
+	"cf4abea7-37b2-11d4-940f-008048da11f9": "attr",        # catalog attribute
+	"45e46cbc-3e24-4165-8b7b-cc98a6f80211": "attr",        # document attribute
+	"888744e1-b616-11d4-9436-004095e12fc7": "column",      # tabular section column
+	"b64d9a41-1642-11d6-a3c7-0050bae0a776": "resource",    # accumulation register
+	"b64d9a42-1642-11d6-a3c7-0050bae0a776": "attr",
+	"b64d9a43-1642-11d6-a3c7-0050bae0a776": "dimension",
+	"13134202-f60b-11d5-a3c7-0050bae0a776": "resource",    # information register
+	"13134203-f60b-11d5-a3c7-0050bae0a776": "dimension",
+	"13134204-f60b-11d5-a3c7-0050bae0a776": "attr",
+}
+
+ROLE_ORDER = ("dimension", "resource", "attr", "column")
+
+
+def is_identity(node):
+	"""["1", ["0","0",<guid>], "<Name>", ["1","ru","<Synonym>"], "", "0", "0"]"""
+	if not (isinstance(node, list) and len(node) >= 3 and isinstance(node[1], list)):
+		return False
+	if len(node[1]) != 3 or not isinstance(node[1][2], str) or not meta.GUID.match(node[1][2]):
+		return False
+	return isinstance(node[2], str) and bool(node[2]) and not node[2].isdigit()
+
+
+def find_identity(node):
+	"""First identity block in a header subtree, depth first."""
+	if not isinstance(node, list):
+		return None
+	if is_identity(node):
+		return node
+	for item in node:
+		found = find_identity(item)
+		if found is not None:
+			return found
+	return None
+
+
+def is_field(node):
+	"""A field is an identity block immediately followed by its type pattern."""
+	return (isinstance(node, list) and len(node) >= 3 and is_identity(node[1])
+	        and isinstance(node[2], list) and node[2] and node[2][0] == "Pattern")
+
+
+def name_of(node):
+	return node[1][2]
+
+
+def guid_of(node):
+	"""The attribute's own GUID, from its identity block.
+
+	This is what DBNames keys its `Fld` entries by, so it is how a ported column is
+	traced back to the `_FldNNN` the data actually sits in.
+	"""
+	ident = node[1]
+	head = ident[1] if len(ident) > 1 else None
+	return head[2] if isinstance(head, list) and len(head) > 2 else None
+
+
+def synonym_of(node):
+	syn = node[1][3] if len(node[1]) > 3 else None
+	return syn[2] if isinstance(syn, list) and len(syn) >= 3 else None
+
+
+def is_collection(node):
+	"""[<class guid>, <count>, member, member, ...] - attributes, dimensions, resources."""
+	return (isinstance(node, list) and len(node) > 2 and isinstance(node[0], str)
+	        and meta.GUID.match(node[0]) and isinstance(node[1], str) and node[1].isdigit())
+
+
+def decode_type(pattern, types):
+	"""['Pattern', ['S','30','1'], guid, ...] -> readable type."""
+	parts, refs = [], []
+	for item in pattern[1:]:
+		if isinstance(item, str) and meta.GUID.match(item):
+			refs.append(item)
+		elif isinstance(item, list) and item:
+			head = item[0]
+			if head == "S":
+				parts.append("string(%s)" % (item[1] if len(item) > 1 else "?"))
+			elif head == "N":
+				parts.append("number(%s,%s)" % (item[1] if len(item) > 1 else "?",
+				                                item[2] if len(item) > 2 else "0"))
+			elif head == "B":
+				parts.append("bool")
+			elif head == "D":
+				parts.append("date(%s)" % (item[1] if len(item) > 1 else "DateTime"))
+			else:
+				refs.extend(s for s in item if isinstance(s, str) and meta.GUID.match(s))
+	for guid in refs:
+		target = types.get(guid)
+		parts.append("-> %s.%s" % (target["family"], target["name"]) if target
+		             else "-> <%s>" % guid[:8])
+	return ", ".join(parts) or "?"
+
+
+def has_pattern(node):
+	if not isinstance(node, list):
+		return False
+	if node and node[0] == "Pattern":
+		return True
+	return any(has_pattern(item) for item in node)
+
+
+def tabular_section_info(node):
+	"""A tabular section is [<header>, ..., <columns collection>]: a named header that
+	carries no type of its own, followed by a collection of typed columns.
+
+	Returns (name, guid). The GUID is what DBNames keys the section's `_VT` table and
+	its `_LineNo` column by.
+	"""
+	if len(node) < 3 or not isinstance(node[0], list) or not is_collection(node[2]):
+		return None
+	if has_pattern(node[0]):
+		return None
+	ident = find_identity(node[0])
+	if ident is None:
+		return None
+	head = ident[1] if len(ident) > 1 else None
+	guid = head[2] if isinstance(head, list) and len(head) > 2 else None
+	return ident[2], guid
+
+
+def tabular_section_name(node):
+	found = tabular_section_info(node)
+	return found[0] if found else None
+
+
+def collect(node, section, collection, types, fields, section_guid=None):
+	if not isinstance(node, list):
+		return
+	if is_field(node):
+		fields.append({
+			"section": section,
+			"sectionGuid": section_guid,
+			"collection": collection,
+			"name": name_of(node),
+			"guid": guid_of(node),
+			"synonym": synonym_of(node),
+			"type": decode_type(node[2], types),
+		})
+		return
+	found = tabular_section_info(node)
+	if found:
+		section, section_guid = found
+	if is_collection(node):
+		collection = node[0]
+		node = node[2:]
+	for item in node:
+		collect(item, section, collection, types, fields, section_guid)
+
+
+def enum_values(tree, object_name):
+	"""Enum members carry no type pattern, so they are plain identity blocks."""
+	values = []
+	def walk(node):
+		if not isinstance(node, list):
+			return
+		if is_identity(node):
+			if node[2] != object_name and node[2] not in values:
+				values.append(node[2])
+			return
+		for item in node:
+			walk(item)
+	walk(tree)
+	return values
+
+
+def load_json(name):
+	with codecs.open(os.path.join(OUT, name), "r", "utf-8") as f:
+		return json.load(f)
+
+
+def fields_of(name, objects, types):
+	guid = next((o["guid"] for o in objects if o.get("name") == name), None)
+	if not guid:
+		return None, None
+	text = meta.load(guid)
+	if text is None:
+		return guid, None
+	fields = []
+	collect(meta.parse(text), None, None, types, fields)
+	return guid, fields
+
+
+def main():
+	objects = load_json("objects.json")
+	types = load_json("typemap.json")
+	report = {r["name"]: r for r in load_json("report.json")}
+
+	if sys.argv[1] == "--model":
+		out_path, names = sys.argv[2], sys.argv[3:]
+		lines = ["# Transport & shipping contour - extracted model", "",
+		         "Generated from the `%s` infobase without 1C software."
+	         % (os.environ.get("DF1C_DB") or os.path.basename(os.environ.get("DF1C_FILE") or "source")), ""]
+		for name in names:
+			guid, fields = fields_of(name, objects, types)
+			row = report.get(name, {})
+			if fields is None:
+				lines += ["## %s *(not found)*" % name, ""]
+				continue
+			lines += ["## %s.%s - %s" % (row.get("family", "?"), name, row.get("synonym") or ""),
+			          "",
+			          "`%s` - %s rows, %s child rows" % (
+				          row.get("physical"),
+				          "{:,}".format(row.get("rows") or 0),
+				          "{:,}".format(row.get("childRows") or 0)), ""]
+			if not fields:
+				values = enum_values(meta.parse(meta.load(guid)), name)
+				if values:
+					lines += ["Values: " + ", ".join(values), ""]
+				continue
+			sections = [None] + sorted({f["section"] for f in fields if f["section"]})
+			for section in sections:
+				chunk = [f for f in fields if f["section"] == section]
+				if not chunk:
+					continue
+				chunk.sort(key=lambda f: ROLE_ORDER.index(ROLE[f["collection"]])
+				           if ROLE.get(f["collection"]) in ROLE_ORDER else 9)
+				lines += ["**%s**" % (section or "header"), "",
+				          "| role | field | synonym | type |", "|---|---|---|---|"]
+				for f in chunk:
+					lines.append("| %s | %s | %s | %s |" % (
+						ROLE.get(f["collection"], "?"), f["name"],
+						f["synonym"] or "", f["type"]))
+				lines.append("")
+		with codecs.open(out_path, "w", "utf-8") as f:
+			f.write("\n".join(lines))
+		print("wrote %s (%d objects)" % (out_path, len(names)))
+		return
+
+	name = sys.argv[1]
+	guid, fields = fields_of(name, objects, types)
+	if fields is None:
+		print("no such object: %s" % name)
+		return
+	print("%s (%s)  fields=%d\n" % (name, guid, len(fields)))
+	for f in fields:
+		print("   %-10s %-22s %-34s %-28s %s"
+		      % (ROLE.get(f["collection"], (f["collection"] or "")[:8]),
+		         f["section"] or "", f["name"], f["synonym"] or "", f["type"]))
+
+
+if __name__ == "__main__":
+	main()