det translator: model DECIMAL-POINT IS COMMA; document the refusals #4462 keeps (#4462)

docs/language_status/det_port_design.md

@@ -408,17 +408,14 @@ Only generator output, never a test case:
   LayoutError / ExprError naming the file, line and character): national / DBCS text, any character beyond Latin-1
   in a name or a literal (estate-crucible KYUY: Kanji names, PIC G, ideographic spaces; KYUYJP had raised
   UnicodeEncodeError), because the translator lays records out and hands the grammar its text one byte a character;
-  a national letter in a name (`BETRÄGE`: the grammar reads ASCII words only); DECIMAL-POINT IS COMMA (`1000,00`,
-  `0,5` must never be read as integers). A national letter inside a literal or a comment is read. #4272: a character
+  a national letter in a name (`BETRÄGE`: the grammar reads ASCII words only). A national letter inside a literal or a comment is read. #4272: a character
   beyond Latin-1 in a `*>` comment is read, and one inside a PROCEDURE DIVISION alphanumeric literal (a UTF-8 em
   dash in a message) makes only its statement a hole by name (`source.narrowed`, gen.WIDE_WHY; oracle_assumptions D4:
   its bytes and length are the source transfer's, not COBOL's), not the program. Still refused, with
   a cause in the cross-check ledger: several programs in one source (PAYMAIN), IDMS (LNIDMS01), and the grammar gaps
   the ledger lists (`translator-refuses-grammar`). `PROGRAM-ID LNCALC.` without its period is read since #4523
-  (`source.logical_lines` puts the period back, as Enterprise COBOL tolerates it). DECIMAL-POINT IS COMMA stays
-  refused: it needs a comma-aware numeric tokenizer in the statement grammar (`MOVE 0,5 TO X` against `A, B`), the
-  layout's VALUE parsing and edited PICTUREs with `.` and `,` swapped, and the runtime's edited moves and DISPLAY;
-  its one program (estate-crucible ZINSBER) would still be refused for its national-letter name `BETRÄGE`.
+  (`source.logical_lines` puts the period back, as Enterprise COBOL tolerates it). DECIMAL-POINT IS COMMA is modelled
+  since the #4462 close-out (below).
 - #4523: the grammar continues a literal only in quotation marks (`'...` at column 72 with `-    '...` on the next
   row did not parse), and reads `""` inside one as two literals (`VALUE "IT""S"` was two values, `MOVE "IT""S"` did
   not parse), though it reads `'IT''S'` whole. `source.as_fixed_rows` now hands the grammar every literal of a
@@ -475,6 +472,32 @@ Only generator output, never a test case:
   text past it and no continuation line ("source defect": fixed-form COBOL reads columns 8-72 only, so
   the program does not compile as written), and a source with no PROGRAM-ID ("not a program": a copybook saved
   with a program's extension, `program_unit`).
+- #4462 close-out. *DECIMAL-POINT IS COMMA* is modelled. `source.comma_literals` hands the grammar each numeric
+  literal's decimal comma as a point, a column for a column (`VALUE 1000,00`, `MOVE 0,5 TO T-WERT (2)`, `-12,5`): a
+  comma between digits is the decimal point, as cobc reads it, and a separator comma is followed by a space; PICTURE
+  strings, literals' text and names keep theirs. A literal written with a decimal point under the clause (`1.5`) is
+  refused by name. An edited PICTURE keeps its characters: `Item.decimal_comma` reads its `,` as the decimal point
+  (scale), and the generated fields carry it to the runtime (`Field.decimalComma()`), where editing
+  (`Editing.format` / `shape`: `.` an insertion character), de-editing, and an alphanumeric sender MOVEd into a
+  numeric or edited receiver (`,` its point, `.` skipped, as libcob reads it) follow it; NUMVAL / NUMVAL-C /
+  TEST-NUMVAL(-C) read their argument with the two swapped. DISPLAY of a numeric item has no decimal point either way.
+  Proven against GnuCOBOL: `test_det_programs.py` DPCOMMA (bytes, typed, groups) -- estate-crucible ZINSBER's
+  statements with its names in ASCII, plus zero suppression, a fixed and a floating sign, check protection, BLANK WHEN
+  ZERO, the alphanumeric moves and NUMVAL / NUMVAL-C. ZINSBER itself now stops at its next gap, the national letter in
+  `GEBÜHR` / `BETRÄGE` (#4664). Found on the way: an edited item compared with a numeric literal is compared as a
+  number by the det runtime and as text by GnuCOBOL (#4665, independent of the clause).
+
+  *Refusals kept deliberately* (each a cause in `tests/cobol_mainframe/fact_crosscheck_ledger.json`, refused by name,
+  never translated on a guess):
+
+  | refusal | programs | why it stays |
+  |---|---|---|
+  | IDMS DML (`translator-refuses-idms`) | estate-crucible LNIDMS01 | BIND RUN-UNIT, OBTAIN, the subschema records have no model in the runtime or the oracle (GnuCOBOL has no IDMS); support is #4532 |
+  | national / DBCS text (`translator-refuses-national-text`) | estate-crucible KYUYJP, KYUYO01, KYUYO02 | the translator lays records out one byte a character; national data is #4272. KYUYJP refuses by name (it had raised UnicodeEncodeError) |
+  | a national letter in a data name (same cause) | estate-crucible ZINSBER | the grammar reads ASCII words only; #4664 |
+  | source defects (`translator-refuses-source-defect`) | GenApp polloo2.cpy | a data entry with no period before the next level number: the source does not compile as written; nothing to fix on either side |
+  | vendor copybooks not in the estate (`translator-refuses-vendor-copybook`) | CardDemo COPAUA0C, COACCT01, CODATE01 (IBM MQ `CMQ*`); CBSA CRECUST (LE `CEEIGZCT`) | the estate ships no such member; the engine records the COPY unresolved, the translator refuses rather than lay out a guessed record |
+  | a COPY of a program member (`translator-copy-program-member`) | estate-crucible ORDMAIN (COPY ORDPRICE), SHPINQ / SHPINQO (COPY SHPRATE) | by design (horror H-0053, its key: "no compiler finds it"): no copy library holds the member, only a program source of that name. Engine copy resolution (#4468) reports the gap; the refusal is correct |
 - #4528: a TS item is the bytes the program wrote "in the region's code page" (CicsTask), and the COBOL side's
   region keeps it in CCSID 037 (cics-crucible SPEC 2; the stub transcodes its Latin-1 storage at the boundary). The
   det port handed CicsTask its storage's bytes (CS, CobolRecords.charset()) as they were: WS-ONE VALUE 'W' reached

gitgalaxy/tools/cobol_to_java/det/cobolrt/Cobol.java

@@ -118,10 +118,10 @@ public static void move(Field from, Field to, Charset cs) {
                 break;
             case NUMERIC:
                 boolean zoned = to.kind == Field.Kind.NUMERIC_DISPLAY && fc != NUMERIC && fc != NUM_EDITED;
-                store(to, zoned ? alnumToDisplay(from.raw(), to, cs) : source(from, fc, cs), cs);
+                store(to, zoned ? alnumToDisplay(from.raw(), to, cs) : source(from, fc, to.decimalComma, cs), cs);
                 break;
             default:
-                edit(to, source(from, fc, cs), cs);
+                edit(to, source(from, fc, to.decimalComma, cs), cs);
         }
     }
 
@@ -250,10 +250,16 @@ private static byte unpunch(byte b, Charset cs) {
 
     /** The sending item's value as a number: numeric, de-edited, or an alphanumeric read as GnuCOBOL does. */
     private static Codec.Num source(Field from, int fc, Charset cs) {
+        return source(from, fc, false, cs);
+    }
+
+    /** As source(from, fc, cs); `decimalComma` (#4462: the receiver's program declares DECIMAL-POINT IS COMMA): an
+     *  alphanumeric sender's `,` is its decimal point and `.` is ignored, as libcob reads it. */
+    private static Codec.Num source(Field from, int fc, boolean decimalComma, Charset cs) {
         switch (fc) {
             case NUMERIC: return Codec.read(from, cs);
             case NUM_EDITED: return deedit(from, cs);
-            default: return parseAlnum(from.raw(), cs);
+            default: return parseAlnum(from.raw(), cs, decimalComma);
         }
     }
 
@@ -287,15 +293,17 @@ private static Codec.Num deedit(Field f, Charset cs) {
     private static Codec.Num alnumToDisplay(byte[] raw, Field to, Charset cs) {
         int total = to.digits;
         int scale = to.scale;
-        if (scale < 0 || scale > total) return parseAlnum(raw, cs);
+        if (scale < 0 || scale > total) return parseAlnum(raw, cs, to.decimalComma);
+        char dp = to.decimalComma ? ',' : '.';  // #4462: DECIMAL-POINT IS COMMA swaps the point and the separator
+        char sep = to.decimalComma ? '.' : ',';
         String s = new String(raw, cs);
         int n = s.length();
         int i = 0;
         while (i < n && Character.isWhitespace(s.charAt(i))) i++;
         boolean neg = false;
         if (i < n && (s.charAt(i) == '+' || s.charAt(i) == '-')) neg = s.charAt(i++) == '-';
         int count = 0;
-        for (int j = i; j < n && s.charAt(j) != '.'; j++) {
+        for (int j = i; j < n && s.charAt(j) != dp; j++) {
             if (s.charAt(j) >= '0' && s.charAt(j) <= '9') count++;
         }
         int size = total - scale;
@@ -315,16 +323,19 @@ private static Codec.Num alnumToDisplay(byte[] raw, Field to, Charset cs) {
             char c = s.charAt(i);
             if (c >= '0' && c <= '9') {
                 out[pos++] = c;
-            } else if (c == '.' && !point) {
+            } else if (c == dp && !point) {
                 point = true;
-            } else if (!(Character.isWhitespace(c) || c == ',')) {
+            } else if (!(Character.isWhitespace(c) || c == sep)) {
                 return new Codec.Num(BigInteger.ZERO, scale, false);
             }
         }
         return new Codec.Num(total == 0 ? BigInteger.ZERO : new BigInteger(new String(out)), scale, neg);
     }
 
-    private static Codec.Num parseAlnum(byte[] raw, Charset cs) {
+    /** `decimalComma` (#4462: DECIMAL-POINT IS COMMA): `,` is the decimal point and `.` the ignored separator. */
+    private static Codec.Num parseAlnum(byte[] raw, Charset cs, boolean decimalComma) {
+        char dp = decimalComma ? ',' : '.';
+        char sep = decimalComma ? '.' : ',';
         String s = new String(raw, cs);
         int i = 0;
         int n = s.length();
@@ -339,9 +350,9 @@ private static Codec.Num parseAlnum(byte[] raw, Charset cs) {
             if (c >= '0' && c <= '9') {
                 digits.append(c);
                 if (point) frac++;
-            } else if (c == '.' && !point) {
+            } else if (c == dp && !point) {
                 point = true;
-            } else if (!(Character.isWhitespace(c) || c == ',')) {
+            } else if (!(Character.isWhitespace(c) || c == sep)) {
                 return new Codec.Num(BigInteger.ZERO, 0, false);
             }
         }
@@ -392,7 +403,7 @@ private static void edit(Field to, Codec.Num n, Charset cs) {
         if (n.neg) v = v.negate();
         String s;
         if (to.blankWhenZero && v.signum() == 0) s = " ".repeat(to.len);
-        else s = Editing.format(to.pic, v, false, null);
+        else s = Editing.format(to.pic, v, to.decimalComma, null);
         padCopy(s.getBytes(cs), to, false, cs);
     }
 
@@ -403,7 +414,7 @@ public static BigDecimal num(Field f, Charset cs) {
         switch (cat(f)) {
             case NUMERIC: return Codec.read(f, cs).value();
             case NUM_EDITED: return deedit(f, cs).value();
-            default: return parseAlnum(f.raw(), cs).value();
+            default: return parseAlnum(f.raw(), cs, f.decimalComma).value();
         }
     }
 

gitgalaxy/tools/cobol_to_java/det/cobolrt/Editing.java

@@ -185,14 +185,19 @@ static String format(String pic, BigDecimal value, boolean decimalComma, String
 
     /** {integer digit positions, decimal digit positions} of an edit PICTURE (floating insertion included). */
     static int[] shape(String pic) {
+        return shape(pic, false);
+    }
+
+    /** As shape(pic); `decimalComma` (#4462: DECIMAL-POINT IS COMMA): `,` is the PICTURE's decimal point. */
+    static int[] shape(String pic, boolean decimalComma) {
         pic = expand(pic);
         char symbol = '$';
         for (int i = 0; i < pic.length(); i++) {
             if (EDITING.indexOf(pic.charAt(i)) == -1) { symbol = pic.charAt(i); break; }
         }
         pic = pic.replace(symbol, '$');
         int decIdx = pic.indexOf('V');
-        if (decIdx == -1) decIdx = pic.indexOf('.');
+        if (decIdx == -1) decIdx = pic.indexOf(decimalComma ? ',' : '.');
         if (decIdx == -1) decIdx = pic.length();
         char floating = 0;
         int floatStart = -1, floatEnd = -1;

gitgalaxy/tools/cobol_to_java/det/cobolrt/Field.java

@@ -20,9 +20,19 @@ public enum Kind {
     final boolean justRight;
     final boolean blankWhenZero;
     final String pic;
+    /** #4462: the program's SPECIAL-NAMES DECIMAL-POINT IS COMMA: `,` is the decimal point of an edited PICTURE and of
+     *  the text an alphanumeric sender MOVEs into this item (GnuCOBOL's module decimal point). */
+    final boolean decimalComma;
 
     private Field(Storage st, int off, int len, Kind kind, int digits, int scale, boolean signed, boolean signLeading,
                   boolean signSeparate, boolean nativeBin, boolean justRight, boolean blankWhenZero, String pic) {
+        this(st, off, len, kind, digits, scale, signed, signLeading, signSeparate, nativeBin, justRight, blankWhenZero,
+                pic, false);
+    }
+
+    private Field(Storage st, int off, int len, Kind kind, int digits, int scale, boolean signed, boolean signLeading,
+                  boolean signSeparate, boolean nativeBin, boolean justRight, boolean blankWhenZero, String pic,
+                  boolean decimalComma) {
         this.st = st;
         this.off = off;
         this.len = len;
@@ -36,6 +46,20 @@ private Field(Storage st, int off, int len, Kind kind, int digits, int scale, bo
         this.justRight = justRight;
         this.blankWhenZero = blankWhenZero;
         this.pic = pic;
+        this.decimalComma = decimalComma;
+    }
+
+    /** #4462: this item in a program that declares DECIMAL-POINT IS COMMA (an edited PICTURE's shape read with `,` as
+     *  its decimal point). */
+    public Field decimalComma() {
+        int d = digits, sc = scale;
+        if (kind == Kind.NUMERIC_EDITED) {
+            int[] shape = Editing.shape(pic, true);
+            d = shape[0] + shape[1];
+            sc = shape[1];
+        }
+        return new Field(st, off, len, kind, d, sc, signed, signLeading, signSeparate, nativeBin, justRight,
+                blankWhenZero, pic, true);
     }
 
     public static Field group(Storage s, int offset, int length) {
@@ -96,14 +120,14 @@ static int binaryLength(int digits) {
     /** A subscript, 1-based: offset + (index - 1) * stride. */
     public Field at(int index, int stride) {
         return new Field(st, off + (index - 1) * stride, len, kind, digits, scale, signed, signLeading, signSeparate,
-                nativeBin, justRight, blankWhenZero, pic);
+                nativeBin, justRight, blankWhenZero, pic, decimalComma);
     }
 
     /** Reference modification (start 1-based; a null length means to the end): an ALPHANUMERIC view. */
     public Field ref(int start, Integer length) {
         int n = length == null ? len - (start - 1) : length;
         return new Field(st, off + start - 1, n, Kind.ALPHANUMERIC, 0, 0, false, false, false, false, false, false,
-                null);
+                null, decimalComma);
     }
 
     public Storage storage() {

gitgalaxy/tools/cobol_to_java/det/gen.py

@@ -754,7 +754,12 @@ def _func(self, f: E.Func) -> str:
         if name == "CURRENT-DATE":
             return "DetCics.currentDate(task.now())" if self.cics is not None else self.clock
         if name in ("NUMVAL", "NUMVAL-C", "TEST-NUMVAL", "TEST-NUMVAL-C") and len(args) == 1:
-            return f"Funcs.{_camel(name)}({self.text(args[0])})"
+            arg = self.text(args[0])
+            if any(r.decimal_comma for r in self.p.records):
+                # #4462: DECIMAL-POINT IS COMMA -- the argument's `,` is its decimal point and `.` its separator:
+                # swapped, it is the text the functions read with the standard ones (a position for a position)
+                arg = f"{arg}.replace('.', '\\u0000').replace(',', '.').replace('\\u0000', ',')"
+            return f"Funcs.{_camel(name)}({arg})"
         if name in ("INTEGER-OF-DATE", "DATE-OF-INTEGER", "INTEGER", "INTEGER-PART", "ABS") and len(args) == 1:
             return f"Funcs.{_camel(name)}({self.num(args[0])})"
         if name in ("MOD", "REM", "MIN", "MAX") and len(args) >= 2:
@@ -1172,6 +1177,12 @@ def _init_one(self, x: L.Item, base: str, top: L.Item, extra: int = 0) -> str:
 
     # ---- fields -------------------------------------------------------------------------------------------------
     def factory(self, it: L.Item, storage: str, offset: str) -> str:
+        f = self._factory(it, storage, offset)
+        # #4462: DECIMAL-POINT IS COMMA -- an edited PICTURE's `,` is its decimal point, and so is an alphanumeric
+        # sender's `,` when the item receives one (the runtime's Field.decimalComma)
+        return f"{f}.decimalComma()" if it.decimal_comma else f
+
+    def _factory(self, it: L.Item, storage: str, offset: str) -> str:
         cat = it.category
         if cat == "GROUP":
             return f"Field.group({storage}, {offset}, {it.size})"

gitgalaxy/tools/cobol_to_java/det/layout.py

@@ -20,6 +20,8 @@
     alphabet_keywords,
     as_fixed_rows,
     cobol_parser,
+    comma_literals,
+    decimal_comma,
     label_records,
     narrowed,
     refusal,
@@ -53,6 +55,7 @@ class Item:
     sign_separate: bool = False
     justified: bool = False
     blank_when_zero: bool = False
+    decimal_comma: bool = False  # #4462: the program's DECIMAL-POINT IS COMMA (an edited PICTURE's `,` is its point)
     children: list = field(default_factory=list)
     conditions: list = field(default_factory=list)  # its 88s
     parent: Item | None = None
@@ -96,7 +99,8 @@ def scale(self) -> int:
         if self.category == "NUMERIC":
             return len(p.split("V", 1)[1]) if "V" in p else -sum(1 for c in p if c == "P")
         if self.category == "NUMERIC-EDITED":
-            dp = p.find(".") if "." in p else p.find("V")
+            point = "," if self.decimal_comma else "."
+            dp = p.find(point) if point in p else p.find("V")
             return sum(1 for c in p[dp + 1 :] if c in "9Z*") if dp >= 0 else 0
         return 0
 
@@ -249,11 +253,12 @@ def _data_only(lines: list[Line]) -> list[Line]:
 
 def parse(lines: list[Line]) -> list[Item]:
     """The 01 / 77 records of the DATA DIVISION, each a tree of Items."""
-    # #4462: national / DBCS text, DECIMAL-POINT IS COMMA, IDMS, several programs (each read on its own): refused by name
+    # #4462: national / DBCS text, IDMS, several programs (each read on its own): refused by name
     why = refusal(lines)  # (a survey's what-if may switch one check off: source.survey_unmask)
     if why:
         raise LayoutError(why)
-    lines = narrowed(lines)  # #4272: a wide character in a `*>` comment / a PROCEDURE DIVISION literal
+    comma = decimal_comma(lines)  # #4462: DECIMAL-POINT IS COMMA (a VALUE's `1000,00`, an edited PICTURE's `,`)
+    lines = comma_literals(narrowed(lines))  # #4272: a wide character in a `*>` comment / a PROCEDURE DIVISION literal
     text, rows = as_fixed_rows(_data_only(lines))
     # the PROCEDURE DIVISION is not needed (and EXEC blocks there are not this grammar's): stop before it
     m = re.search(r"^ {7}\s*PROCEDURE\s+DIVISION\b", text, re.I | re.M)
@@ -317,6 +322,8 @@ def records_append(node, section, fd):
             raise LayoutError(missing)
         raise LayoutError(f"DATA DIVISION does not parse near expanded line(s) {errors[:5]}")
     for r in records:
+        for it in r.walk():
+            it.decimal_comma = comma
         _inherit_usage(r, None)
         layout(r)
     # an 01 REDEFINES another 01 of its section shares that record's storage