diff --git a/docs/language_status/det_port_design.md b/docs/language_status/det_port_design.md index f8ffce40f..1fe99fe2e 100644 --- a/docs/language_status/det_port_design.md +++ b/docs/language_status/det_port_design.md @@ -408,17 +408,14 @@ Only generator output, never a test case: LayoutError / ExprError naming the file, line and character): national / DBCS text, any character beyond Latin-1 in a name or a literal (estate-crucible KYUY: Kanji names, PIC G, ideographic spaces; KYUYJP had raised UnicodeEncodeError), because the translator lays records out and hands the grammar its text one byte a character; - a national letter in a name (`BETRÄGE`: the grammar reads ASCII words only); DECIMAL-POINT IS COMMA (`1000,00`, - `0,5` must never be read as integers). A national letter inside a literal or a comment is read. #4272: a character + a national letter in a name (`BETRÄGE`: the grammar reads ASCII words only). A national letter inside a literal or a comment is read. #4272: a character beyond Latin-1 in a `*>` comment is read, and one inside a PROCEDURE DIVISION alphanumeric literal (a UTF-8 em dash in a message) makes only its statement a hole by name (`source.narrowed`, gen.WIDE_WHY; oracle_assumptions D4: its bytes and length are the source transfer's, not COBOL's), not the program. Still refused, with a cause in the cross-check ledger: several programs in one source (PAYMAIN), IDMS (LNIDMS01), and the grammar gaps the ledger lists (`translator-refuses-grammar`). `PROGRAM-ID LNCALC.` without its period is read since #4523 - (`source.logical_lines` puts the period back, as Enterprise COBOL tolerates it). DECIMAL-POINT IS COMMA stays - refused: it needs a comma-aware numeric tokenizer in the statement grammar (`MOVE 0,5 TO X` against `A, B`), the - layout's VALUE parsing and edited PICTUREs with `.` and `,` swapped, and the runtime's edited moves and DISPLAY; - its one program (estate-crucible ZINSBER) would still be refused for its national-letter name `BETRÄGE`. + (`source.logical_lines` puts the period back, as Enterprise COBOL tolerates it). DECIMAL-POINT IS COMMA is modelled + since the #4462 close-out (below). - #4523: the grammar continues a literal only in quotation marks (`'...` at column 72 with `- '...` on the next row did not parse), and reads `""` inside one as two literals (`VALUE "IT""S"` was two values, `MOVE "IT""S"` did not parse), though it reads `'IT''S'` whole. `source.as_fixed_rows` now hands the grammar every literal of a @@ -475,6 +472,32 @@ Only generator output, never a test case: text past it and no continuation line ("source defect": fixed-form COBOL reads columns 8-72 only, so the program does not compile as written), and a source with no PROGRAM-ID ("not a program": a copybook saved with a program's extension, `program_unit`). +- #4462 close-out. *DECIMAL-POINT IS COMMA* is modelled. `source.comma_literals` hands the grammar each numeric + literal's decimal comma as a point, a column for a column (`VALUE 1000,00`, `MOVE 0,5 TO T-WERT (2)`, `-12,5`): a + comma between digits is the decimal point, as cobc reads it, and a separator comma is followed by a space; PICTURE + strings, literals' text and names keep theirs. A literal written with a decimal point under the clause (`1.5`) is + refused by name. An edited PICTURE keeps its characters: `Item.decimal_comma` reads its `,` as the decimal point + (scale), and the generated fields carry it to the runtime (`Field.decimalComma()`), where editing + (`Editing.format` / `shape`: `.` an insertion character), de-editing, and an alphanumeric sender MOVEd into a + numeric or edited receiver (`,` its point, `.` skipped, as libcob reads it) follow it; NUMVAL / NUMVAL-C / + TEST-NUMVAL(-C) read their argument with the two swapped. DISPLAY of a numeric item has no decimal point either way. + Proven against GnuCOBOL: `test_det_programs.py` DPCOMMA (bytes, typed, groups) -- estate-crucible ZINSBER's + statements with its names in ASCII, plus zero suppression, a fixed and a floating sign, check protection, BLANK WHEN + ZERO, the alphanumeric moves and NUMVAL / NUMVAL-C. ZINSBER itself now stops at its next gap, the national letter in + `GEBÜHR` / `BETRÄGE` (#4664). Found on the way: an edited item compared with a numeric literal is compared as a + number by the det runtime and as text by GnuCOBOL (#4665, independent of the clause). + + *Refusals kept deliberately* (each a cause in `tests/cobol_mainframe/fact_crosscheck_ledger.json`, refused by name, + never translated on a guess): + + | refusal | programs | why it stays | + |---|---|---| + | IDMS DML (`translator-refuses-idms`) | estate-crucible LNIDMS01 | BIND RUN-UNIT, OBTAIN, the subschema records have no model in the runtime or the oracle (GnuCOBOL has no IDMS); support is #4532 | + | national / DBCS text (`translator-refuses-national-text`) | estate-crucible KYUYJP, KYUYO01, KYUYO02 | the translator lays records out one byte a character; national data is #4272. KYUYJP refuses by name (it had raised UnicodeEncodeError) | + | a national letter in a data name (same cause) | estate-crucible ZINSBER | the grammar reads ASCII words only; #4664 | + | source defects (`translator-refuses-source-defect`) | GenApp polloo2.cpy | a data entry with no period before the next level number: the source does not compile as written; nothing to fix on either side | + | vendor copybooks not in the estate (`translator-refuses-vendor-copybook`) | CardDemo COPAUA0C, COACCT01, CODATE01 (IBM MQ `CMQ*`); CBSA CRECUST (LE `CEEIGZCT`) | the estate ships no such member; the engine records the COPY unresolved, the translator refuses rather than lay out a guessed record | + | a COPY of a program member (`translator-copy-program-member`) | estate-crucible ORDMAIN (COPY ORDPRICE), SHPINQ / SHPINQO (COPY SHPRATE) | by design (horror H-0053, its key: "no compiler finds it"): no copy library holds the member, only a program source of that name. Engine copy resolution (#4468) reports the gap; the refusal is correct | - #4528: a TS item is the bytes the program wrote "in the region's code page" (CicsTask), and the COBOL side's region keeps it in CCSID 037 (cics-crucible SPEC 2; the stub transcodes its Latin-1 storage at the boundary). The det port handed CicsTask its storage's bytes (CS, CobolRecords.charset()) as they were: WS-ONE VALUE 'W' reached diff --git a/gitgalaxy/tools/cobol_to_java/det/cobolrt/Cobol.java b/gitgalaxy/tools/cobol_to_java/det/cobolrt/Cobol.java index 0362511cc..6217d3fb7 100644 --- a/gitgalaxy/tools/cobol_to_java/det/cobolrt/Cobol.java +++ b/gitgalaxy/tools/cobol_to_java/det/cobolrt/Cobol.java @@ -118,10 +118,10 @@ public static void move(Field from, Field to, Charset cs) { break; case NUMERIC: boolean zoned = to.kind == Field.Kind.NUMERIC_DISPLAY && fc != NUMERIC && fc != NUM_EDITED; - store(to, zoned ? alnumToDisplay(from.raw(), to, cs) : source(from, fc, cs), cs); + store(to, zoned ? alnumToDisplay(from.raw(), to, cs) : source(from, fc, to.decimalComma, cs), cs); break; default: - edit(to, source(from, fc, cs), cs); + edit(to, source(from, fc, to.decimalComma, cs), cs); } } @@ -250,10 +250,16 @@ private static byte unpunch(byte b, Charset cs) { /** The sending item's value as a number: numeric, de-edited, or an alphanumeric read as GnuCOBOL does. */ private static Codec.Num source(Field from, int fc, Charset cs) { + return source(from, fc, false, cs); + } + + /** As source(from, fc, cs); `decimalComma` (#4462: the receiver's program declares DECIMAL-POINT IS COMMA): an + * alphanumeric sender's `,` is its decimal point and `.` is ignored, as libcob reads it. */ + private static Codec.Num source(Field from, int fc, boolean decimalComma, Charset cs) { switch (fc) { case NUMERIC: return Codec.read(from, cs); case NUM_EDITED: return deedit(from, cs); - default: return parseAlnum(from.raw(), cs); + default: return parseAlnum(from.raw(), cs, decimalComma); } } @@ -287,7 +293,9 @@ private static Codec.Num deedit(Field f, Charset cs) { private static Codec.Num alnumToDisplay(byte[] raw, Field to, Charset cs) { int total = to.digits; int scale = to.scale; - if (scale < 0 || scale > total) return parseAlnum(raw, cs); + if (scale < 0 || scale > total) return parseAlnum(raw, cs, to.decimalComma); + char dp = to.decimalComma ? ',' : '.'; // #4462: DECIMAL-POINT IS COMMA swaps the point and the separator + char sep = to.decimalComma ? '.' : ','; String s = new String(raw, cs); int n = s.length(); int i = 0; @@ -295,7 +303,7 @@ private static Codec.Num alnumToDisplay(byte[] raw, Field to, Charset cs) { boolean neg = false; if (i < n && (s.charAt(i) == '+' || s.charAt(i) == '-')) neg = s.charAt(i++) == '-'; int count = 0; - for (int j = i; j < n && s.charAt(j) != '.'; j++) { + for (int j = i; j < n && s.charAt(j) != dp; j++) { if (s.charAt(j) >= '0' && s.charAt(j) <= '9') count++; } int size = total - scale; @@ -315,16 +323,19 @@ private static Codec.Num alnumToDisplay(byte[] raw, Field to, Charset cs) { char c = s.charAt(i); if (c >= '0' && c <= '9') { out[pos++] = c; - } else if (c == '.' && !point) { + } else if (c == dp && !point) { point = true; - } else if (!(Character.isWhitespace(c) || c == ',')) { + } else if (!(Character.isWhitespace(c) || c == sep)) { return new Codec.Num(BigInteger.ZERO, scale, false); } } return new Codec.Num(total == 0 ? BigInteger.ZERO : new BigInteger(new String(out)), scale, neg); } - private static Codec.Num parseAlnum(byte[] raw, Charset cs) { + /** `decimalComma` (#4462: DECIMAL-POINT IS COMMA): `,` is the decimal point and `.` the ignored separator. */ + private static Codec.Num parseAlnum(byte[] raw, Charset cs, boolean decimalComma) { + char dp = decimalComma ? ',' : '.'; + char sep = decimalComma ? '.' : ','; String s = new String(raw, cs); int i = 0; int n = s.length(); @@ -339,9 +350,9 @@ private static Codec.Num parseAlnum(byte[] raw, Charset cs) { if (c >= '0' && c <= '9') { digits.append(c); if (point) frac++; - } else if (c == '.' && !point) { + } else if (c == dp && !point) { point = true; - } else if (!(Character.isWhitespace(c) || c == ',')) { + } else if (!(Character.isWhitespace(c) || c == sep)) { return new Codec.Num(BigInteger.ZERO, 0, false); } } @@ -392,7 +403,7 @@ private static void edit(Field to, Codec.Num n, Charset cs) { if (n.neg) v = v.negate(); String s; if (to.blankWhenZero && v.signum() == 0) s = " ".repeat(to.len); - else s = Editing.format(to.pic, v, false, null); + else s = Editing.format(to.pic, v, to.decimalComma, null); padCopy(s.getBytes(cs), to, false, cs); } @@ -403,7 +414,7 @@ public static BigDecimal num(Field f, Charset cs) { switch (cat(f)) { case NUMERIC: return Codec.read(f, cs).value(); case NUM_EDITED: return deedit(f, cs).value(); - default: return parseAlnum(f.raw(), cs).value(); + default: return parseAlnum(f.raw(), cs, f.decimalComma).value(); } } diff --git a/gitgalaxy/tools/cobol_to_java/det/cobolrt/Editing.java b/gitgalaxy/tools/cobol_to_java/det/cobolrt/Editing.java index 196e90d6c..02ebbf512 100644 --- a/gitgalaxy/tools/cobol_to_java/det/cobolrt/Editing.java +++ b/gitgalaxy/tools/cobol_to_java/det/cobolrt/Editing.java @@ -185,6 +185,11 @@ static String format(String pic, BigDecimal value, boolean decimalComma, String /** {integer digit positions, decimal digit positions} of an edit PICTURE (floating insertion included). */ static int[] shape(String pic) { + return shape(pic, false); + } + + /** As shape(pic); `decimalComma` (#4462: DECIMAL-POINT IS COMMA): `,` is the PICTURE's decimal point. */ + static int[] shape(String pic, boolean decimalComma) { pic = expand(pic); char symbol = '$'; for (int i = 0; i < pic.length(); i++) { @@ -192,7 +197,7 @@ static int[] shape(String pic) { } pic = pic.replace(symbol, '$'); int decIdx = pic.indexOf('V'); - if (decIdx == -1) decIdx = pic.indexOf('.'); + if (decIdx == -1) decIdx = pic.indexOf(decimalComma ? ',' : '.'); if (decIdx == -1) decIdx = pic.length(); char floating = 0; int floatStart = -1, floatEnd = -1; diff --git a/gitgalaxy/tools/cobol_to_java/det/cobolrt/Field.java b/gitgalaxy/tools/cobol_to_java/det/cobolrt/Field.java index b0cf89e05..5221c2f10 100644 --- a/gitgalaxy/tools/cobol_to_java/det/cobolrt/Field.java +++ b/gitgalaxy/tools/cobol_to_java/det/cobolrt/Field.java @@ -20,9 +20,19 @@ public enum Kind { final boolean justRight; final boolean blankWhenZero; final String pic; + /** #4462: the program's SPECIAL-NAMES DECIMAL-POINT IS COMMA: `,` is the decimal point of an edited PICTURE and of + * the text an alphanumeric sender MOVEs into this item (GnuCOBOL's module decimal point). */ + final boolean decimalComma; private Field(Storage st, int off, int len, Kind kind, int digits, int scale, boolean signed, boolean signLeading, boolean signSeparate, boolean nativeBin, boolean justRight, boolean blankWhenZero, String pic) { + this(st, off, len, kind, digits, scale, signed, signLeading, signSeparate, nativeBin, justRight, blankWhenZero, + pic, false); + } + + private Field(Storage st, int off, int len, Kind kind, int digits, int scale, boolean signed, boolean signLeading, + boolean signSeparate, boolean nativeBin, boolean justRight, boolean blankWhenZero, String pic, + boolean decimalComma) { this.st = st; this.off = off; this.len = len; @@ -36,6 +46,20 @@ private Field(Storage st, int off, int len, Kind kind, int digits, int scale, bo this.justRight = justRight; this.blankWhenZero = blankWhenZero; this.pic = pic; + this.decimalComma = decimalComma; + } + + /** #4462: this item in a program that declares DECIMAL-POINT IS COMMA (an edited PICTURE's shape read with `,` as + * its decimal point). */ + public Field decimalComma() { + int d = digits, sc = scale; + if (kind == Kind.NUMERIC_EDITED) { + int[] shape = Editing.shape(pic, true); + d = shape[0] + shape[1]; + sc = shape[1]; + } + return new Field(st, off, len, kind, d, sc, signed, signLeading, signSeparate, nativeBin, justRight, + blankWhenZero, pic, true); } public static Field group(Storage s, int offset, int length) { @@ -96,14 +120,14 @@ static int binaryLength(int digits) { /** A subscript, 1-based: offset + (index - 1) * stride. */ public Field at(int index, int stride) { return new Field(st, off + (index - 1) * stride, len, kind, digits, scale, signed, signLeading, signSeparate, - nativeBin, justRight, blankWhenZero, pic); + nativeBin, justRight, blankWhenZero, pic, decimalComma); } /** Reference modification (start 1-based; a null length means to the end): an ALPHANUMERIC view. */ public Field ref(int start, Integer length) { int n = length == null ? len - (start - 1) : length; return new Field(st, off + start - 1, n, Kind.ALPHANUMERIC, 0, 0, false, false, false, false, false, false, - null); + null, decimalComma); } public Storage storage() { diff --git a/gitgalaxy/tools/cobol_to_java/det/gen.py b/gitgalaxy/tools/cobol_to_java/det/gen.py index a8c850245..91be8ce64 100644 --- a/gitgalaxy/tools/cobol_to_java/det/gen.py +++ b/gitgalaxy/tools/cobol_to_java/det/gen.py @@ -754,7 +754,12 @@ def _func(self, f: E.Func) -> str: if name == "CURRENT-DATE": return "DetCics.currentDate(task.now())" if self.cics is not None else self.clock if name in ("NUMVAL", "NUMVAL-C", "TEST-NUMVAL", "TEST-NUMVAL-C") and len(args) == 1: - return f"Funcs.{_camel(name)}({self.text(args[0])})" + arg = self.text(args[0]) + if any(r.decimal_comma for r in self.p.records): + # #4462: DECIMAL-POINT IS COMMA -- the argument's `,` is its decimal point and `.` its separator: + # swapped, it is the text the functions read with the standard ones (a position for a position) + arg = f"{arg}.replace('.', '\\u0000').replace(',', '.').replace('\\u0000', ',')" + return f"Funcs.{_camel(name)}({arg})" if name in ("INTEGER-OF-DATE", "DATE-OF-INTEGER", "INTEGER", "INTEGER-PART", "ABS") and len(args) == 1: return f"Funcs.{_camel(name)}({self.num(args[0])})" if name in ("MOD", "REM", "MIN", "MAX") and len(args) >= 2: @@ -1172,6 +1177,12 @@ def _init_one(self, x: L.Item, base: str, top: L.Item, extra: int = 0) -> str: # ---- fields ------------------------------------------------------------------------------------------------- def factory(self, it: L.Item, storage: str, offset: str) -> str: + f = self._factory(it, storage, offset) + # #4462: DECIMAL-POINT IS COMMA -- an edited PICTURE's `,` is its decimal point, and so is an alphanumeric + # sender's `,` when the item receives one (the runtime's Field.decimalComma) + return f"{f}.decimalComma()" if it.decimal_comma else f + + def _factory(self, it: L.Item, storage: str, offset: str) -> str: cat = it.category if cat == "GROUP": return f"Field.group({storage}, {offset}, {it.size})" diff --git a/gitgalaxy/tools/cobol_to_java/det/layout.py b/gitgalaxy/tools/cobol_to_java/det/layout.py index 32c0d7363..745b7735e 100644 --- a/gitgalaxy/tools/cobol_to_java/det/layout.py +++ b/gitgalaxy/tools/cobol_to_java/det/layout.py @@ -20,6 +20,8 @@ alphabet_keywords, as_fixed_rows, cobol_parser, + comma_literals, + decimal_comma, label_records, narrowed, refusal, @@ -53,6 +55,7 @@ class Item: sign_separate: bool = False justified: bool = False blank_when_zero: bool = False + decimal_comma: bool = False # #4462: the program's DECIMAL-POINT IS COMMA (an edited PICTURE's `,` is its point) children: list = field(default_factory=list) conditions: list = field(default_factory=list) # its 88s parent: Item | None = None @@ -96,7 +99,8 @@ def scale(self) -> int: if self.category == "NUMERIC": return len(p.split("V", 1)[1]) if "V" in p else -sum(1 for c in p if c == "P") if self.category == "NUMERIC-EDITED": - dp = p.find(".") if "." in p else p.find("V") + point = "," if self.decimal_comma else "." + dp = p.find(point) if point in p else p.find("V") return sum(1 for c in p[dp + 1 :] if c in "9Z*") if dp >= 0 else 0 return 0 @@ -249,11 +253,12 @@ def _data_only(lines: list[Line]) -> list[Line]: def parse(lines: list[Line]) -> list[Item]: """The 01 / 77 records of the DATA DIVISION, each a tree of Items.""" - # #4462: national / DBCS text, DECIMAL-POINT IS COMMA, IDMS, several programs (each read on its own): refused by name + # #4462: national / DBCS text, IDMS, several programs (each read on its own): refused by name why = refusal(lines) # (a survey's what-if may switch one check off: source.survey_unmask) if why: raise LayoutError(why) - lines = narrowed(lines) # #4272: a wide character in a `*>` comment / a PROCEDURE DIVISION literal + comma = decimal_comma(lines) # #4462: DECIMAL-POINT IS COMMA (a VALUE's `1000,00`, an edited PICTURE's `,`) + lines = comma_literals(narrowed(lines)) # #4272: a wide character in a `*>` comment / a PROCEDURE DIVISION literal text, rows = as_fixed_rows(_data_only(lines)) # the PROCEDURE DIVISION is not needed (and EXEC blocks there are not this grammar's): stop before it m = re.search(r"^ {7}\s*PROCEDURE\s+DIVISION\b", text, re.I | re.M) @@ -317,6 +322,8 @@ def records_append(node, section, fd): raise LayoutError(missing) raise LayoutError(f"DATA DIVISION does not parse near expanded line(s) {errors[:5]}") for r in records: + for it in r.walk(): + it.decimal_comma = comma _inherit_usage(r, None) layout(r) # an 01 REDEFINES another 01 of its section shares that record's storage diff --git a/gitgalaxy/tools/cobol_to_java/det/source.py b/gitgalaxy/tools/cobol_to_java/det/source.py index d37a3afac..d381bdcad 100644 --- a/gitgalaxy/tools/cobol_to_java/det/source.py +++ b/gitgalaxy/tools/cobol_to_java/det/source.py @@ -49,8 +49,8 @@ class Line: UNMASKABLE = { "cut-literal": "a literal cut open at column 72 (cut_literal): read with its text past column 72, as if fixed", "several-programs": "several programs in one source (several_programs): read as one", - "unmodelled": "national / DBCS text, a national letter in a name, IDMS, DECIMAL-POINT IS COMMA, the stand-in " - "control characters (unmodelled): read as the grammar is handed them", + "unmodelled": "national / DBCS text, a national letter in a name, IDMS, a decimal-point literal under " + "DECIMAL-POINT IS COMMA, the stand-in control characters (unmodelled): read as the grammar is handed them", "missing-copybook": "a COPY found in no directory: expanded to nothing, so its items are `no such item` holes", } _unmasked: frozenset[str] = frozenset() @@ -340,6 +340,12 @@ def cut_literal(lines: list[Line]) -> str | None: _DECIMAL_COMMA = re.compile(r"\bDECIMAL-POINT\s+(?:IS\s+)?COMMA\b", re.I) +# #4462: a PICTURE character-string (its `.` and `,` are editing symbols, never a literal's decimal point) +_PICTURE = re.compile(r"(? str | None: character. It had raised UnicodeEncodeError; a DBCS estate is refused, never laid out wrong. - A national letter in a word outside a literal (`BETRÄGE`, read in cp273): the COBOL grammar reads ASCII words only, and refused the line unnamed. - - DECIMAL-POINT IS COMMA (DEUT ZINSBER): `1000,00` and `0,5` are numbers and an edited PIC's `.` and `,` swap - roles; not modelled, so never read as a list of integers. + - #4462: under DECIMAL-POINT IS COMMA (modelled: comma_literals, Item.decimal_comma), a numeric literal written + with a decimal POINT (`1.5`): the clause makes `,` the decimal point, so it is no number the compiler reads. - #4462: IDMS (IDMS-CONTROL SECTION, SCHEMA SECTION; estate-crucible LOAN LNIDMS01): its DML (BIND RUN-UNIT, READY, OBTAIN CALC, FINISH, DC RETURN) and subschema records are not modelled, so the program is refused by name (#4532 tracks IDMS support), never parsed as COBOL with holes. @@ -384,6 +390,7 @@ def unmodelled(lines: list[Line]) -> str | None: The IDENTIFICATION DIVISION's paragraphs after PROGRAM-ID (AUTHOR, REMARKS ...) are free text no parser reads.""" in_id = in_proc = False + comma = decimal_comma(lines) for ln in lines: head = ln.text.lstrip().upper() if re.match(r"(?:IDENTIFICATION|ID)\s+DIVISION\b", head): @@ -406,11 +413,48 @@ def unmodelled(lines: list[Line]) -> str | None: "ASCII words only") # fmt: skip if _IDMS.search(bare): return f"{Path(ln.file).name}:{ln.line}: IDMS DML not supported (an IDMS-DC / DMLC program: {bare.strip()})" - if _DECIMAL_COMMA.search(bare): - return f"{Path(ln.file).name}:{ln.line}: DECIMAL-POINT IS COMMA is not modelled" + point = _POINT_POINT.search(_numeric_text(ln.text)) if comma else None + if point is not None: + return (f"{Path(ln.file).name}:{ln.line}: the literal {point.group(0)} has a decimal point under " + "DECIMAL-POINT IS COMMA: not modelled") # fmt: skip return None +def decimal_comma(lines: list[Line]) -> bool: + """#4462: whether the program declares SPECIAL-NAMES DECIMAL-POINT IS COMMA (before its DATA DIVISION).""" + for ln in lines: + if re.match(r"(?:DATA|PROCEDURE)\s+DIVISION\b", ln.text.lstrip(), re.I): + return False + if _DECIMAL_COMMA.search(_outside_literals(ln.text)): + return True + return False + + +def _numeric_text(text: str) -> str: + """`text` with its literals' content, its `*>` comment and its PICTURE strings blanked (a column for a column): + where a numeric literal can be.""" + bare = _outside_literals(text[: _comment_at(text)]).ljust(len(text)) + return _PICTURE.sub(lambda m: " " * len(m.group(0)), bare) + + +def comma_literals(lines: list[Line]) -> list[Line]: + """#4462: under DECIMAL-POINT IS COMMA (estate-crucible DEUT ZINSBER), each numeric literal's decimal comma handed + to the grammar as a point, a column for a column (`1000,00` -> `1000.00`, `0,5` -> `0.5`), so a VALUE and a + PROCEDURE DIVISION literal are the numbers cobc reads (a comma between digits is the decimal point; a separator + comma is followed by a space), never a list of integers. A PICTURE keeps its characters as written: its `,` is + the decimal point there (layout.Item.decimal_comma, and the runtime's Field.decimalComma). Without the clause, + `lines` unchanged.""" + if not decimal_comma(lines): + return lines + out: list[Line] = [] + for ln in lines: + text = list(ln.text) + for m in _COMMA_POINT.finditer(_numeric_text(ln.text)): + text[m.start(1)] = "." + out.append(Line("".join(text), ln.file, ln.line, ln.cut)) + return out + + def _comment_at(text: str) -> int: """Where `text`'s `*>` comment starts (one outside a literal), or len(text).""" at = text.find("*>") diff --git a/gitgalaxy/tools/cobol_to_java/det/stmt.py b/gitgalaxy/tools/cobol_to_java/det/stmt.py index 3de8517d6..0a06f2c6c 100644 --- a/gitgalaxy/tools/cobol_to_java/det/stmt.py +++ b/gitgalaxy/tools/cobol_to_java/det/stmt.py @@ -20,6 +20,7 @@ _outside_literals, as_fixed_rows, cobol_parser, + comma_literals, narrowed, refusal, unwrap, @@ -72,11 +73,12 @@ def __init__(self, kind: str, node: Stmt | None, target: list): def parse(lines: list[Line]) -> Procedure: parser = _parser_cache() # first: a missing translator extra fails here, before any work - # #4462: national / DBCS text, DECIMAL-POINT IS COMMA, IDMS, several programs (each read on its own): refused by name + # #4462: national / DBCS text, IDMS, several programs (each read on its own): refused by name why = refusal(lines) # (a survey's what-if may switch one check off: source.survey_unmask) if why: raise E.ExprError(why) - lines = narrowed(lines) # #4272: a wide character in a `*>` comment / a PROCEDURE DIVISION literal + # #4272: a wide character in a `*>` comment / a PROCEDURE DIVISION literal; #4462: a decimal comma's literal + lines = comma_literals(narrowed(lines)) text, rows = as_fixed_rows(lines) m = re.search(r"^ {7}\s*PROCEDURE\s+DIVISION\b[^.]*\.", text, re.I | re.M) if not m: diff --git a/tests/cobol_mainframe/fact_crosscheck_ledger.json b/tests/cobol_mainframe/fact_crosscheck_ledger.json index 985880b05..2d72074f6 100644 --- a/tests/cobol_mainframe/fact_crosscheck_ledger.json +++ b/tests/cobol_mainframe/fact_crosscheck_ledger.json @@ -28,9 +28,9 @@ "summary": "POINTER is 4 bytes on z/OS (the engine, the keys) and 8 in the translator's storage, GnuCOBOL on x86-64 (oracle_assumptions.md C9): every field after one moves by 4. Both right for their platform." }, "translator-copy-program-member": { - "side": "translator", + "side": "definitional", "issue": 4460, - "summary": "COPY SHPRATE / COPY ORDPRICE find only a program source. Before #4468, det.source spliced that whole program in. Since #4468 the engine's gap (no declared library holds the member) refuses the program by name instead (CopyUnresolved)." + "summary": "COPY SHPRATE / COPY ORDPRICE: by design no copy library of the estate holds the member, only a program source of that name (estate-crucible horror H-0053; its key: 'no compiler finds it'). The engine records the gap and the translator refuses the program by name (CopyUnresolved, #4468; before #4468 det.source spliced the program in, #4460). A correct refusal on both sides." }, "translator-refuses-idms": { "side": "translator", @@ -40,7 +40,7 @@ "translator-refuses-national-text": { "side": "translator", "issue": 4462, - "summary": "Refused by name since #4462 (source.unmodelled), never laid out wrong: national / DBCS text (Kanji names and literals, PIC G, ideographic spaces: estate-crucible KYUY, decoded in its declared cp930 / shift_jis or UTF-8; the translator lays records out one byte a character), and DECIMAL-POINT IS COMMA (DEUT ZINSBER: 1000,00 / 0,5 are numbers). Modelling either is a translator feature." + "summary": "Refused by name since #4462 (source.unmodelled), never laid out wrong: national / DBCS text (Kanji names and literals, PIC G, ideographic spaces: estate-crucible KYUY, decoded in its declared cp930 / shift_jis or UTF-8; the translator lays records out one byte a character; #4272), and a single-byte national letter in a data name (DEUT ZINSBER's GEBÜHR / BETRÄGE in cp273: the grammar reads ASCII words only; #4664). ZINSBER's DECIMAL-POINT IS COMMA is modelled since #4462 (test_det_programs DPCOMMA)." }, "translator-refuses-source-defect": { "side": "definitional", @@ -331,7 +331,7 @@ "dsf :: unit_extents | src/GML/PLUKKFRN.cobol | translator | (procedure division) L73-75": "engine-main-line-no-extent", "dsf :: value | src/GML/FO04D1X1.cobol | engine | L106 I2 = 001": "numeric-value-spelling", "dsf :: value | src/GML/FO04D1X1.cobol | translator | L106 I2 = 1": "numeric-value-spelling", - "estate-crucible :: status | apps/DEUT/cobol/ZINSBER.cbl | engine | translator refused: layout: LayoutError: ZINSBER.cbl:12: DECIMAL-POINT IS COMMA is not modelled; procedure: ExprError: ZINSBER.cbl:12: DECIMAL-POINT IS COMMA is not modelled": "translator-refuses-national-text", + "estate-crucible :: status | apps/DEUT/cobol/ZINSBER.cbl | engine | translator refused: layout: LayoutError: ZINSSATZ.cpy:3: the name GEBÜHR holds a national letter: the COBOL grammar reads ASCII words only; procedure: ExprError: ZINSSATZ.cpy:3: the name GEBÜHR holds a national letter: the COBOL grammar reads ASCII words only": "translator-refuses-national-text", "estate-crucible :: status | apps/KYUY/cobol/KYUYJP.cbl | engine | translator refused: layout: LayoutError: KYUYJP.cbl:2: national / DBCS text ('給', U+7D66) is not modelled: the translator reads a single-byte code page; procedure: ExprError: KYUYJP.cbl:2: national / DBCS text ('給', U+7D66) is not modelled: the translator reads a single-byte code page": "translator-refuses-national-text", "estate-crucible :: status | apps/KYUY/cobol/KYUYO01.cbl | engine | translator refused: layout: LayoutError: KYUYO01.cbl:10: national / DBCS text ('処', U+51E6) is not modelled: the translator reads a single-byte code page; procedure: ExprError: KYUYO01.cbl:10: national / DBCS text ('処', U+51E6) is not modelled: the translator reads a single-byte code page": "translator-refuses-national-text", "estate-crucible :: status | apps/KYUY/cobol/KYUYO02.cbl | engine | translator refused: layout: LayoutError: KYUYO02.cbl:8: national / DBCS text ('賞', U+8CDE) is not modelled: the translator reads a single-byte code page; procedure: ExprError: KYUYO02.cbl:8: national / DBCS text ('賞', U+8CDE) is not modelled: the translator reads a single-byte code page": "translator-refuses-national-text", diff --git a/tests/cobol_mainframe/test_det_programs.py b/tests/cobol_mainframe/test_det_programs.py index e5f33ae63..a59fdc54b 100644 --- a/tests/cobol_mainframe/test_det_programs.py +++ b/tests/cobol_mainframe/test_det_programs.py @@ -980,6 +980,44 @@ def _all_data(items) -> list[str]: _all_proc(_ALL_LETTERS, ["ALL 'x'", "ALL 'ab'", "ALL SPACES"]), ) + +def dpc_program(name: str, data: list[str], proc: list[str]) -> str: + """#4462: a program under SPECIAL-NAMES DECIMAL-POINT IS COMMA: WORKING-STORAGE and PROCEDURE DIVISION lines from + column 8 (a statement indented four more).""" + lines = ["IDENTIFICATION DIVISION.", f"PROGRAM-ID. {name}.", "ENVIRONMENT DIVISION.", "CONFIGURATION SECTION.", + "SPECIAL-NAMES.", " DECIMAL-POINT IS COMMA.", "DATA DIVISION.", "WORKING-STORAGE SECTION.", *data, + "PROCEDURE DIVISION.", *[f" {x}" for x in proc], " GOBACK."] # fmt: skip + return "\n".join(f" {x}" for x in lines) + "\n" + + +# #4462: DECIMAL-POINT IS COMMA (estate-crucible DEUT ZINSBER's statements, its national-letter names spelt in ASCII): +# numeric literals with a decimal comma in VALUE clauses and statements (`1000,00`, `0,5`, `-12,5`, a subscript +# beside one), numeric-edited PICTUREs whose `,` is the decimal point and `.` an insertion character (zero +# suppression, a fixed and a floating sign, check protection, BLANK WHEN ZERO), de-editing (an edited item MOVEd as +# a number), an alphanumeric sender into a zoned, a packed and an edited item (`,` its decimal point, `.` +# skipped, as libcob reads it) and NUMVAL / NUMVAL-C (`,` the point, `.` the separator) +PROGRAMS["DPCOMMA"] = dpc_program( + "DPCOMMA", + ["01 WS-ZINS.", " 05 ZINS-SATZ PIC 9V99 VALUE 1,50.", " 05 GEBUEHR PIC 9(3)V99 VALUE 12,50.", + "01 BETRAEGE.", " 05 BETRAG PIC 9(7)V99 VALUE 1000,00.", " 05 ERGEBNIS PIC ZZZ.ZZ9,99.", + " 05 TABELLE OCCURS 3 TIMES.", " 10 T-WERT PIC 9V9.", + "01 S-WERT PIC S9(3)V9 VALUE -12,5.", "01 E-SIGN PIC -Z.ZZ9,9.", "01 E-FLOAT PIC +++.++9,99.", + "01 E-STAR PIC **.**9,99.", "01 E-FIX PIC 9(3),9(2).", "01 E-BWZ PIC ZZ9,99 BLANK WHEN ZERO.", + "01 N-ZON PIC 9(5)V99.", "01 N-PAK PIC S9(5)V99 COMP-3.", "01 N-OUT PIC 9(7)V99.", + "01 X-TXT PIC X(10) VALUE '12,34'.", "01 X-DOT PIC X(10) VALUE '1.234,5'."], + ["COMPUTE BETRAG = BETRAG * ZINS-SATZ", "MOVE 0,5 TO T-WERT (2)", "MOVE BETRAG TO ERGEBNIS", "DISPLAY ERGEBNIS", + "DISPLAY BETRAEGE", "MOVE S-WERT TO E-SIGN", "DISPLAY E-SIGN", "MOVE -1234,56 TO E-SIGN", "DISPLAY E-SIGN", + "MOVE 1234,5 TO E-FLOAT", "DISPLAY E-FLOAT", "MOVE -0,05 TO E-FLOAT", "DISPLAY E-FLOAT", + "MOVE 12,3 TO E-STAR", "DISPLAY E-STAR", "MOVE 12,3 TO E-FIX", "DISPLAY E-FIX", "MOVE ZERO TO E-BWZ", + "DISPLAY '[' E-BWZ ']'", "MOVE 7,5 TO E-BWZ", "DISPLAY E-BWZ", "MOVE ERGEBNIS TO N-ZON", "DISPLAY N-ZON", + "COMPUTE N-OUT = BETRAG + GEBUEHR - S-WERT", "DISPLAY N-OUT", + "MOVE X-TXT TO N-ZON", "DISPLAY N-ZON", "MOVE X-TXT TO N-PAK", "MOVE N-PAK TO N-OUT", "DISPLAY N-OUT", + "MOVE X-DOT TO N-ZON", "DISPLAY N-ZON", "MOVE X-TXT TO ERGEBNIS", "DISPLAY ERGEBNIS", + "COMPUTE N-ZON = FUNCTION NUMVAL('7,25')", "DISPLAY N-ZON", + "COMPUTE N-ZON = FUNCTION NUMVAL-C('1.234,50')", "DISPLAY N-ZON"], +) # fmt: skip + + FILE_DDS = {"SORTUG": ["INFILE", "OUTFILE", "M1FILE", "M2FILE", "MGFILE"]} diff --git a/tests/cobol_mainframe/test_det_source_4462.py b/tests/cobol_mainframe/test_det_source_4462.py index 18f0b1072..f6fdda6df 100644 --- a/tests/cobol_mainframe/test_det_source_4462.py +++ b/tests/cobol_mainframe/test_det_source_4462.py @@ -1,7 +1,7 @@ """#4462: what the det translator reads before it parses -- the code page the estate declares (estate-crucible `key/manifest.json` `code_pages`, which the engine reads through `--source-encoding`), free-format source, and the -text it refuses by name instead of crashing (national / DBCS text, a national letter in a name, DECIMAL-POINT IS -COMMA).""" +text it refuses by name instead of crashing (national / DBCS text, a national letter in a name), and DECIMAL-POINT IS +COMMA (read; a literal written with a decimal point under it refused by name).""" from __future__ import annotations @@ -106,8 +106,8 @@ def test_engine_copies_carry_the_pages_from_the_ir_and_the_port_ticket(tmp_path) (("01 BETRÄGE PIC 9(3).", "PROCEDURE DIVISION.", " GOBACK."), r"PROG\.cbl:5: the name BETRÄGE holds a national letter"), (("01 B PIC X(4).", "PROCEDURE DIVISION.", " MOVE 'Ä' TO GEBÜHR", " GOBACK."), r"the name GEBÜHR"), - # ZINSBER: `VALUE 1000,00` / `MOVE 0,5`: never read as integers - (("01 B PIC 9(7)V99 VALUE 1000,00.", "PROCEDURE DIVISION.", " MOVE 0,5 TO B", " GOBACK."), None), + # under DECIMAL-POINT IS COMMA, `1.5` is no number the compiler reads (`1,5` is: test_decimal_point_is_comma_*) + (("01 B PIC 9(7)V99 VALUE 1000,00.", "PROCEDURE DIVISION.", " MOVE 1.5 TO B", " GOBACK."), None), ], ) # fmt: skip def test_text_the_translator_cannot_read_is_refused_by_name(body, why): @@ -121,13 +121,62 @@ def test_text_the_translator_cannot_read_is_refused_by_name(body, why): lines = _lines("01 B PIC X.")[:2] + [SRC.Line(t, "/x/PROG.cbl", 0) for t in ( "ENVIRONMENT DIVISION.", "CONFIGURATION SECTION.", "SPECIAL-NAMES.", " DECIMAL-POINT IS COMMA.")] + \ _lines(*body)[2:] # fmt: skip - why = "DECIMAL-POINT IS COMMA is not modelled" + why = r"PROG\.cbl:7: the literal 1\.5 has a decimal point under DECIMAL-POINT IS COMMA: not modelled" with pytest.raises(L.LayoutError, match=why): L.parse(lines) with pytest.raises(E.ExprError, match=why): ST.parse(lines) +def _dpc(*body: str) -> list: + """A program under SPECIAL-NAMES DECIMAL-POINT IS COMMA (estate-crucible DEUT ZINSBER's clause).""" + head = ( + "IDENTIFICATION DIVISION.", + "PROGRAM-ID. PROG.", + "ENVIRONMENT DIVISION.", + "CONFIGURATION SECTION.", + "SPECIAL-NAMES.", + " DECIMAL-POINT IS COMMA.", + "DATA DIVISION.", + "WORKING-STORAGE SECTION.", + ) + return [SRC.Line(t, "/x/PROG.cbl", n) for n, t in enumerate((*head, *body), 1)] + + +def test_decimal_point_is_comma_hands_the_grammar_each_literal_with_a_point(): + """#4462: a comma between digits is a numeric literal's decimal point (a column for a column); a separator comma + (followed by a space), a PICTURE, a literal's text and a name keep theirs. Without the clause nothing changes.""" + body = ("01 B PIC 9(7)V99 VALUE 1000,00.", "01 E PIC ZZZ.ZZ9,99.", "01 F PIC 99,99 VALUE '1,5'.", + "01 T.", " 05 W PIC 9V9 OCCURS 3.", "PROCEDURE DIVISION.", " MOVE 0,5 TO W (2)", + " COMPUTE B = -12,5 * B, 3", " CALL 'X' USING A-1,5 B", " GOBACK.") # fmt: skip + got = [ln.text for ln in SRC.comma_literals(_dpc(*body))][8:] + assert got == ["01 B PIC 9(7)V99 VALUE 1000.00.", "01 E PIC ZZZ.ZZ9,99.", "01 F PIC 99,99 VALUE '1,5'.", + "01 T.", " 05 W PIC 9V9 OCCURS 3.", "PROCEDURE DIVISION.", " MOVE 0.5 TO W (2)", + " COMPUTE B = -12.5 * B, 3", " CALL 'X' USING A-1,5 B", " GOBACK."] # fmt: skip + assert all(len(a.text) == len(b.text) for a, b in zip(SRC.comma_literals(_dpc(*body)), _dpc(*body))) + plain = _lines(*body) + assert SRC.comma_literals(plain) is plain and not SRC.decimal_comma(plain) + + +def test_decimal_point_is_comma_is_read_by_layout_and_statements(): + """#4462 (estate-crucible ZINSBER, its names in ASCII): VALUE 1000,00 is 1000.00 and `MOVE 0,5` moves 0.5; an + edited PICTURE's `,` is its decimal point (scale), its `.` an insertion character.""" + pytest.importorskip("tree_sitter_language_pack") + from decimal import Decimal + + from gitgalaxy.tools.cobol_to_java.det import layout as L + from gitgalaxy.tools.cobol_to_java.det import stmt as ST + + lines = _dpc("01 BETRAG PIC 9(7)V99 VALUE 1000,00.", "01 ERGEBNIS PIC ZZZ.ZZ9,99.", "01 T.", + " 05 W PIC 9V9 OCCURS 3.", "PROCEDURE DIVISION.", " MOVE 0,5 TO W (2)", " GOBACK.") # fmt: skip + recs = {r.name: r for r in L.parse(lines)} + assert recs["BETRAG"].values == [("num", Decimal("1000.00"))] and L.image(recs["BETRAG"]) == b"000100000" + e = recs["ERGEBNIS"] + assert (e.category, e.size, e.scale, e.decimal_comma) == ("NUMERIC-EDITED", 10, 2, True) + move = ST.parse(lines).paragraphs[0].body[0] + assert move.kind == "MOVE" and "0.5" in str(move.data), move.data + + def test_national_text_in_a_comment_or_a_latin1_literal_is_read(): pytest.importorskip("tree_sitter_language_pack") from gitgalaxy.tools.cobol_to_java.det import layout as L