From 12fb416fc9a0002b79e1123c390d46ab67bfc5af Mon Sep 17 00:00:00 2001 From: Kevin Day Date: Sat, 20 Jun 2026 17:08:46 -0500 Subject: [PATCH] Progress: Various updates. The main FSS specification with clarifications. I'm generally in favor of performing calculations once. However, I decided to try reducing the memory allocation and deallocation cost of variables by applying the simple math operations each time for some functions where this is performed only once. This has a cost of some code readability, however. The affected functions are single purpose and simple making the cost rather moot. Begin changing the logic to add a new feature to solve a problem with insufficient compliance to the Unicode standards. The joiners need to be handled. The combining combines into what is effectively a graph character. Treat the joiner characters the same, even though one can join two non-printing characters. New lines must ignore joiners (I have seen this as the standard behavior by a lot of software and decided to go with that to avoid problems). The is_combining, is_joiner, and is_combining_joiner are new features added to help solve these problems. A lot of the logic is incomplete as of this commit. I didn't want to add a cost of auto-checking every character after the first. This means that I cannot have the is_space, is_graph, or other such functions automnate this. Therefore, the logic must be changed to check for these spaces. The code will become more complex but this will not incur as much of a performance cost compared to checking each character 2x+ times. I also decided not to continue on after the first combining and joiner characters. The combining already makes something a graph. The joiner requires more searched and is highly subjective. Therefore, make life easier and just treat all joiners as graph-making for the purposes of processing and syntax. --- build/stand_alone/byte_dump.config.h | 2 + build/stand_alone/example.config.h | 2 + build/stand_alone/fake.config.h | 2 + build/stand_alone/fss_identify.config.h | 4 +- build/stand_alone/fss_read.config.fss_basic.h | 2 + .../fss_read.config.fss_basic_list.h | 2 + .../fss_read.config.fss_embedded_list.h | 2 + .../fss_read.config.fss_extended.h | 2 + .../fss_read.config.fss_extended_list.h | 2 + .../stand_alone/fss_read.config.fss_payload.h | 2 + build/stand_alone/fss_read.config.h | 2 + .../stand_alone/fss_write.config.fss_basic.h | 2 + .../fss_write.config.fss_basic_list.h | 2 + .../fss_write.config.fss_embedded_list.h | 2 + .../fss_write.config.fss_extended.h | 2 + .../fss_write.config.fss_extended_list.h | 2 + .../fss_write.config.fss_payload.h | 2 + build/stand_alone/fss_write.config.h | 2 + build/stand_alone/iki_read.config.eki.h | 2 + build/stand_alone/iki_read.config.h | 4 +- build/stand_alone/iki_write.config.eki.h | 4 +- build/stand_alone/iki_write.config.h | 4 +- build/stand_alone/status_code.config.fss.h | 2 + build/stand_alone/status_code.config.h | 2 + build/stand_alone/utf8.config.h | 2 + level_0/f_fss/c/fss.c | 280 +++++++++++------- level_0/f_fss/c/fss.h | 47 +-- level_0/f_fss/data/build/settings-tests | 2 +- .../documentation/man/man3/f_fss_is_graph.3 | 6 +- .../documentation/man/man3/f_fss_is_space.3 | 4 +- .../f_fss/tests/unit/c/test-fss-is_space.c | 22 +- .../unit/c/test-fss-skip_past_combining.c | 183 ++++++++++++ .../unit/c/test-fss-skip_past_combining.h | 34 +++ .../tests/unit/c/test-fss-skip_past_delimit.c | 21 +- .../tests/unit/c/test-fss-skip_past_space.c | 65 +++- level_0/f_fss/tests/unit/c/test-fss.c | 3 + level_0/f_fss/tests/unit/c/test-fss.h | 1 + level_0/f_parse/c/parse.c | 8 +- level_0/f_utf/c/private-utf_combining.c | 4 +- level_0/f_utf/c/private-utf_combining.h | 6 +- level_0/f_utf/c/private-utf_joiner.c | 4 +- level_0/f_utf/c/private-utf_joiner.h | 6 +- level_0/f_utf/c/private-utf_zero_width.c | 10 + level_0/f_utf/c/utf.c | 2 +- level_0/f_utf/c/utf/is.c | 27 ++ level_0/f_utf/c/utf/is.h | 23 ++ level_0/f_utf/c/utf/is_character.c | 16 + level_0/f_utf/c/utf/is_character.h | 17 ++ level_0/f_utf/data/build/settings-tests | 1 + level_0/f_utf/tests/unit/c/data-utf.c | 5 + level_0/f_utf/tests/unit/c/data-utf.h | 15 + .../test-utf-character_is_combining_joiner.c | 66 +++++ .../test-utf-character_is_combining_joiner.h | 20 ++ .../unit/c/test-utf-is_combining_joiner.c | 100 +++++++ .../unit/c/test-utf-is_combining_joiner.h | 20 ++ level_0/f_utf/tests/unit/c/test-utf.c | 2 + level_0/f_utf/tests/unit/c/test-utf.h | 2 + level_1/fl_fss/c/fss/basic.c | 3 +- level_1/fl_fss/c/fss/basic.h | 8 + level_1/fl_fss/c/fss/basic_list.h | 8 + level_1/fl_fss/c/fss/embedded_list.h | 8 + level_1/fl_fss/c/fss/extended.c | 3 +- level_1/fl_fss/c/fss/extended.h | 8 + level_1/fl_fss/c/fss/extended_list.c | 94 ++++-- level_1/fl_fss/c/fss/extended_list.h | 8 + level_1/fl_fss/c/private-fss.c | 48 +-- level_1/fl_fss/c/private-fss.h | 4 + .../man/man3/fl_fss_basic_content_read.3 | 2 + .../man/man3/fl_fss_basic_content_write.3 | 2 + .../man/man3/fl_fss_basic_list_content_read.3 | 2 + .../man3/fl_fss_basic_list_content_write.3 | 2 + .../man/man3/fl_fss_basic_list_object_read.3 | 2 + .../man/man3/fl_fss_basic_list_object_write.3 | 2 + .../man/man3/fl_fss_basic_object_read.3 | 2 + .../man/man3/fl_fss_basic_object_write.3 | 2 + .../man3/fl_fss_embedded_list_content_read.3 | 2 + .../man3/fl_fss_embedded_list_content_write.3 | 2 + .../man3/fl_fss_embedded_list_object_read.3 | 2 + .../man3/fl_fss_embedded_list_object_write.3 | 2 + .../man/man3/fl_fss_extended_content_read.3 | 2 + .../man/man3/fl_fss_extended_content_write.3 | 2 + .../man3/fl_fss_extended_list_content_read.3 | 2 + .../man3/fl_fss_extended_list_content_write.3 | 2 + .../man3/fl_fss_extended_list_object_read.3 | 2 + .../man3/fl_fss_extended_list_object_write.3 | 2 + .../man/man3/fl_fss_extended_object_read.3 | 2 + .../man/man3/fl_fss_extended_object_write.3 | 2 + specifications/fss.txt | 18 +- 88 files changed, 1099 insertions(+), 231 deletions(-) create mode 100644 level_0/f_fss/tests/unit/c/test-fss-skip_past_combining.c create mode 100644 level_0/f_fss/tests/unit/c/test-fss-skip_past_combining.h create mode 100644 level_0/f_utf/tests/unit/c/test-utf-character_is_combining_joiner.c create mode 100644 level_0/f_utf/tests/unit/c/test-utf-character_is_combining_joiner.h create mode 100644 level_0/f_utf/tests/unit/c/test-utf-is_combining_joiner.c create mode 100644 level_0/f_utf/tests/unit/c/test-utf-is_combining_joiner.h diff --git a/build/stand_alone/byte_dump.config.h b/build/stand_alone/byte_dump.config.h index 083ae5ae7..3cb3dc3b6 100644 --- a/build/stand_alone/byte_dump.config.h +++ b/build/stand_alone/byte_dump.config.h @@ -1331,6 +1331,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ //#define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ //#define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1368,6 +1369,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/example.config.h b/build/stand_alone/example.config.h index 58390029c..85d7859cb 100644 --- a/build/stand_alone/example.config.h +++ b/build/stand_alone/example.config.h @@ -1296,6 +1296,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1333,6 +1334,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fake.config.h b/build/stand_alone/fake.config.h index 83146cb51..45cc3e753 100644 --- a/build/stand_alone/fake.config.h +++ b/build/stand_alone/fake.config.h @@ -1774,6 +1774,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1811,6 +1812,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_identify.config.h b/build/stand_alone/fss_identify.config.h index 45b667a06..6f2f88849 100644 --- a/build/stand_alone/fss_identify.config.h +++ b/build/stand_alone/fss_identify.config.h @@ -1477,6 +1477,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1514,6 +1515,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ @@ -1522,7 +1524,7 @@ #define _di_f_utf_is_emoji_ #define _di_f_utf_is_fragment_ #define _di_f_utf_is_graph_ -//#define _di_f_utf_is_joiner_ +#define _di_f_utf_is_joiner_ #define _di_f_utf_is_numeric_ #define _di_f_utf_is_phonetic_ #define _di_f_utf_is_private_ diff --git a/build/stand_alone/fss_read.config.fss_basic.h b/build/stand_alone/fss_read.config.fss_basic.h index 9f9f46d65..2526178cf 100644 --- a/build/stand_alone/fss_read.config.fss_basic.h +++ b/build/stand_alone/fss_read.config.fss_basic.h @@ -1548,6 +1548,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1585,6 +1586,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_read.config.fss_basic_list.h b/build/stand_alone/fss_read.config.fss_basic_list.h index 850d757f1..4f9e105d2 100644 --- a/build/stand_alone/fss_read.config.fss_basic_list.h +++ b/build/stand_alone/fss_read.config.fss_basic_list.h @@ -1548,6 +1548,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1585,6 +1586,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_read.config.fss_embedded_list.h b/build/stand_alone/fss_read.config.fss_embedded_list.h index e36e2d776..65624b403 100644 --- a/build/stand_alone/fss_read.config.fss_embedded_list.h +++ b/build/stand_alone/fss_read.config.fss_embedded_list.h @@ -1548,6 +1548,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1585,6 +1586,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_read.config.fss_extended.h b/build/stand_alone/fss_read.config.fss_extended.h index 4b0131ed3..ced1f3d32 100644 --- a/build/stand_alone/fss_read.config.fss_extended.h +++ b/build/stand_alone/fss_read.config.fss_extended.h @@ -1548,6 +1548,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1585,6 +1586,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_read.config.fss_extended_list.h b/build/stand_alone/fss_read.config.fss_extended_list.h index bcd22d7c4..8b045020d 100644 --- a/build/stand_alone/fss_read.config.fss_extended_list.h +++ b/build/stand_alone/fss_read.config.fss_extended_list.h @@ -1548,6 +1548,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1585,6 +1586,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_read.config.fss_payload.h b/build/stand_alone/fss_read.config.fss_payload.h index 60cb9625f..18bfdec07 100644 --- a/build/stand_alone/fss_read.config.fss_payload.h +++ b/build/stand_alone/fss_read.config.fss_payload.h @@ -1548,6 +1548,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1585,6 +1586,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_read.config.h b/build/stand_alone/fss_read.config.h index 2ad930395..991feb200 100644 --- a/build/stand_alone/fss_read.config.h +++ b/build/stand_alone/fss_read.config.h @@ -1548,6 +1548,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1585,6 +1586,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_write.config.fss_basic.h b/build/stand_alone/fss_write.config.fss_basic.h index f054c635c..34e03ce2b 100644 --- a/build/stand_alone/fss_write.config.fss_basic.h +++ b/build/stand_alone/fss_write.config.fss_basic.h @@ -1491,6 +1491,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1528,6 +1529,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_write.config.fss_basic_list.h b/build/stand_alone/fss_write.config.fss_basic_list.h index 48f29fc5b..3132f4b83 100644 --- a/build/stand_alone/fss_write.config.fss_basic_list.h +++ b/build/stand_alone/fss_write.config.fss_basic_list.h @@ -1491,6 +1491,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1528,6 +1529,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_write.config.fss_embedded_list.h b/build/stand_alone/fss_write.config.fss_embedded_list.h index 4553b0fbd..04d8e9956 100644 --- a/build/stand_alone/fss_write.config.fss_embedded_list.h +++ b/build/stand_alone/fss_write.config.fss_embedded_list.h @@ -1491,6 +1491,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1528,6 +1529,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_write.config.fss_extended.h b/build/stand_alone/fss_write.config.fss_extended.h index 9ab47c305..215ce9705 100644 --- a/build/stand_alone/fss_write.config.fss_extended.h +++ b/build/stand_alone/fss_write.config.fss_extended.h @@ -1491,6 +1491,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1528,6 +1529,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_write.config.fss_extended_list.h b/build/stand_alone/fss_write.config.fss_extended_list.h index 1c84e7488..57536a753 100644 --- a/build/stand_alone/fss_write.config.fss_extended_list.h +++ b/build/stand_alone/fss_write.config.fss_extended_list.h @@ -1491,6 +1491,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1528,6 +1529,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_write.config.fss_payload.h b/build/stand_alone/fss_write.config.fss_payload.h index b6ba40d16..ce360aafe 100644 --- a/build/stand_alone/fss_write.config.fss_payload.h +++ b/build/stand_alone/fss_write.config.fss_payload.h @@ -1491,6 +1491,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1528,6 +1529,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/fss_write.config.h b/build/stand_alone/fss_write.config.h index 1391ed714..406054a13 100644 --- a/build/stand_alone/fss_write.config.h +++ b/build/stand_alone/fss_write.config.h @@ -1491,6 +1491,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1528,6 +1529,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/iki_read.config.eki.h b/build/stand_alone/iki_read.config.eki.h index c97dc831c..9d18e3936 100644 --- a/build/stand_alone/iki_read.config.eki.h +++ b/build/stand_alone/iki_read.config.eki.h @@ -1392,6 +1392,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1429,6 +1430,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/iki_read.config.h b/build/stand_alone/iki_read.config.h index ed6d4d2e6..b6756f88e 100644 --- a/build/stand_alone/iki_read.config.h +++ b/build/stand_alone/iki_read.config.h @@ -1392,6 +1392,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1429,6 +1430,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ @@ -1437,7 +1439,7 @@ #define _di_f_utf_is_emoji_ #define _di_f_utf_is_fragment_ #define _di_f_utf_is_graph_ -//#define _di_f_utf_is_joiner_ +#define _di_f_utf_is_joiner_ #define _di_f_utf_is_numeric_ #define _di_f_utf_is_phonetic_ #define _di_f_utf_is_private_ diff --git a/build/stand_alone/iki_write.config.eki.h b/build/stand_alone/iki_write.config.eki.h index e8f879256..02e837647 100644 --- a/build/stand_alone/iki_write.config.eki.h +++ b/build/stand_alone/iki_write.config.eki.h @@ -1343,6 +1343,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1380,6 +1381,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ @@ -1388,7 +1390,7 @@ #define _di_f_utf_is_emoji_ #define _di_f_utf_is_fragment_ #define _di_f_utf_is_graph_ -//#define _di_f_utf_is_joiner_ +#define _di_f_utf_is_joiner_ #define _di_f_utf_is_numeric_ #define _di_f_utf_is_phonetic_ #define _di_f_utf_is_private_ diff --git a/build/stand_alone/iki_write.config.h b/build/stand_alone/iki_write.config.h index 81ee6982d..01ea4f280 100644 --- a/build/stand_alone/iki_write.config.h +++ b/build/stand_alone/iki_write.config.h @@ -1343,6 +1343,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1380,6 +1381,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ @@ -1388,7 +1390,7 @@ #define _di_f_utf_is_emoji_ #define _di_f_utf_is_fragment_ #define _di_f_utf_is_graph_ -//#define _di_f_utf_is_joiner_ +#define _di_f_utf_is_joiner_ #define _di_f_utf_is_numeric_ #define _di_f_utf_is_phonetic_ #define _di_f_utf_is_private_ diff --git a/build/stand_alone/status_code.config.fss.h b/build/stand_alone/status_code.config.fss.h index 63c04cfc6..a07a44486 100644 --- a/build/stand_alone/status_code.config.fss.h +++ b/build/stand_alone/status_code.config.fss.h @@ -1461,6 +1461,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1498,6 +1499,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/status_code.config.h b/build/stand_alone/status_code.config.h index 5f4bcb3b0..ea8486fb4 100644 --- a/build/stand_alone/status_code.config.h +++ b/build/stand_alone/status_code.config.h @@ -1391,6 +1391,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ #define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ #define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1428,6 +1429,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/build/stand_alone/utf8.config.h b/build/stand_alone/utf8.config.h index b7a5d024b..65a188020 100644 --- a/build/stand_alone/utf8.config.h +++ b/build/stand_alone/utf8.config.h @@ -1349,6 +1349,7 @@ #define _di_f_utf_character_is_alphabetic_numeric_ #define _di_f_utf_character_is_ascii_ //#define _di_f_utf_character_is_combining_ +#define _di_f_utf_character_is_combining_joinger_ //#define _di_f_utf_character_is_control_ #define _di_f_utf_character_is_control_code_ #define _di_f_utf_character_is_control_format_ @@ -1386,6 +1387,7 @@ #define _di_f_utf_is_alphabetic_numeric_ #define _di_f_utf_is_ascii_ //#define _di_f_utf_is_combining_ +#define _di_f_utf_is_combining_joiner_ //#define _di_f_utf_is_control_ #define _di_f_utf_is_control_code_ #define _di_f_utf_is_control_format_ diff --git a/level_0/f_fss/c/fss.c b/level_0/f_fss/c/fss.c index df7bff748..335dde1e1 100644 --- a/level_0/f_fss/c/fss.c +++ b/level_0/f_fss/c/fss.c @@ -164,13 +164,13 @@ extern "C" { return F_false; } - f_number_unsigned_t width_max = (range.stop - range.start) + 1; + state->status = f_utf_is_combining( + buffer.string + range.start, + (range.stop - range.start) + 1 > buffer.used - range.start + ? buffer.used - range.start + : (range.stop - range.start) + 1 + ); - if (width_max > buffer.used - range.start) { - width_max = buffer.used - range.start; - } - - state->status = f_utf_is_combining(buffer.string + range.start, width_max); if (state->status == F_true) return F_true; f_fss_fail_utf_to_false(state); @@ -191,13 +191,13 @@ extern "C" { return F_false; } - f_number_unsigned_t width_max = (range.stop - range.start) + 1; - - if (width_max > buffer.used - range.start) { - width_max = buffer.used - range.start; - } + state->status = f_utf_is_graph( + buffer.string + range.start, + (range.stop - range.start) + 1 > buffer.used - range.start + ? buffer.used - range.start + : (range.stop - range.start) + 1 + ); - state->status = f_utf_is_graph(buffer.string + range.start, width_max); if (state->status == F_true) return F_true; f_fss_fail_utf_to_false(state); @@ -218,29 +218,56 @@ extern "C" { return F_false; } - f_number_unsigned_t width_max = (range.stop - range.start) + 1; + state->status = f_utf_is_zero_width( + buffer.string + range.start, + (range.stop - range.start) + 1 > buffer.used - range.start + ? buffer.used - range.start + : (range.stop - range.start) + 1 + ); - if (width_max > buffer.used - range.start) { - width_max = buffer.used - range.start; + if (state->status == F_true) { + state->status = F_space; + + return F_false; } - state->status = f_utf_is_zero_width(buffer.string + range.start, width_max); - if (state->status != F_false) return F_false; + if (state->status == F_false) { + state->status = f_utf_is_whitespace( + buffer.string + range.start, + (range.stop - range.start) + 1 > buffer.used - range.start + ? buffer.used - range.start + : (range.stop - range.start) + 1, + F_false + ); + + if (state->status == F_false) return F_false; + } - f_fss_fail_utf_to_false(state); + if (state->status == F_true) { + state->status = f_utf_is_control( + buffer.string + range.start, + (range.stop - range.start) + 1 > buffer.used - range.start + ? buffer.used - range.start + : (range.stop - range.start) + 1 + ); + + if (state->status == F_true) { + state->status = F_control; + + // Some ASCII control characters are space characters. + if (!macro_f_utf_byte_width_is(buffer.string[range.start])) { + return F_true; + } - state->status = f_utf_is_whitespace(buffer.string + range.start, width_max, F_false); - if (state->status == F_true) return F_true; + return F_false; + } + else if (state->status == F_false) { + return F_true; + } + } f_fss_fail_utf_to_false(state); - if (state->status == F_false) { - state->status = f_utf_is_control(buffer.string + range.start, width_max); - if (state->status == F_true) return F_true; - - f_fss_fail_utf_to_false(state); - } - return F_false; } #endif // _di_f_fss_is_space_ @@ -257,13 +284,13 @@ extern "C" { return F_false; } - f_number_unsigned_t width_max = (range.stop - range.start) + 1; + state->status = f_utf_is_zero_width( + buffer.string + range.start, + (range.stop - range.start) + 1 > buffer.used - range.start + ? buffer.used - range.start + : (range.stop - range.start) + 1 + ); - if (width_max > buffer.used - range.start) { - width_max = buffer.used - range.start; - } - - state->status = f_utf_is_zero_width(buffer.string + range.start, width_max); if (state->status == F_true) return F_true; f_fss_fail_utf_to_false(state); @@ -347,49 +374,26 @@ extern "C" { if (buffer.string[range->start] != f_fss_placeholder_s.string[0]) break; } // for - uint8_t width = 0; - f_number_unsigned_t width_max = (range->stop - range->start) + 1; + if (macro_f_utf_byte_width_is(buffer.string[range->start])) { + if (range->start + macro_f_utf_byte_width(buffer.string[range->start]) > buffer.used) { + state->status = F_status_set_error(F_complete_not_utf_eos); - if (width_max > buffer.used - range->start) { - width_max = buffer.used - range->start; - } + f_fss_fail_utf(state); - // Check that the first character is not a combining character. - state->status = f_utf_is_combining(buffer.string + range->start, width_max); + return; + } - f_fss_fail_utf_to_false(state); - if (F_status_is_error(state->status)) return; + if (range->start + (macro_f_utf_byte_width(buffer.string[range->start]) - 1) > range->stop) { + state->status = F_status_set_error(F_complete_not_utf_stop); - if (state->status == F_true) { - state->status = F_is_not; + f_fss_fail_utf(state); - return; + return; + } } for (;;) { - width = macro_f_utf_byte_width(buffer.string[range->start]); - - if (width > 1) { - if (range->start + width >= buffer.used) { - state->status = F_status_set_error(F_complete_not_utf_eos); - - f_fss_fail_utf(state); - - return; - } - - if (range->start + width > range->stop) { - state->status = F_status_set_error(F_complete_not_utf_stop); - - f_fss_fail_utf(state); - - return; - } - } - - range->start += width; - if (range->start >= buffer.used) { state->status = F_okay_eos; @@ -402,12 +406,6 @@ extern "C" { return; } - width_max = (range->stop - range->start) + 1; - - if (width_max > buffer.used - range->start) { - width_max = buffer.used - range->start; - } - if (buffer.string[range->start] == f_fss_eol_s.string[0]) { state->status = F_okay_eol; @@ -420,12 +418,37 @@ extern "C" { continue; } - state->status = f_utf_is_combining(buffer.string + range->start, width_max); + state->status = f_utf_is_combining_joiner( + buffer.string + range->start, + (range->stop - range->start) + 1 > buffer.used - range->start + ? buffer.used - range->start + : (range->stop - range->start) + 1 + ); f_fss_fail_utf_to_false(state); if (state->status == F_false) break; if (F_status_is_error(state->status)) return; + + if (macro_f_utf_byte_width_is(buffer.string[range->start])) { + if (range->start + macro_f_utf_byte_width(buffer.string[range->start]) > buffer.used) { + state->status = F_status_set_error(F_complete_not_utf_eos); + + f_fss_fail_utf(state); + + return; + } + + if (range->start + (macro_f_utf_byte_width(buffer.string[range->start]) - 1) > range->stop) { + state->status = F_status_set_error(F_complete_not_utf_stop); + + f_fss_fail_utf(state); + + return; + } + } + + range->start += macro_f_utf_byte_width(buffer.string[range->start]); } // for state->status = F_okay; @@ -507,18 +530,36 @@ extern "C" { if (buffer.string[range->start] != f_fss_placeholder_s.string[0]) break; } // for - uint8_t width = 0; f_number_unsigned_t previous = range->start; - f_number_unsigned_t width_max = (range->stop - range->start) + 1; - if (width_max > buffer.used - range->start) { - width_max = buffer.used - range->start; + if (macro_f_utf_byte_width_is(buffer.string[range->start])) { + if (range->start + macro_f_utf_byte_width(buffer.string[range->start]) > buffer.used) { + state->status = F_status_set_error(F_complete_not_utf_eos); + + f_fss_fail_utf(state); + + return; + } + + if (range->start + (macro_f_utf_byte_width(buffer.string[range->start]) - 1) > range->stop) { + state->status = F_status_set_error(F_complete_not_utf_stop); + + f_fss_fail_utf(state); + + return; + } } - // Check that the first character is not a combining character. - state->status = f_utf_is_combining(buffer.string + range->start, width_max); + // Check that the first character is not a combining character or a joiner. + state->status = f_utf_is_combining_joiner( + buffer.string + previous, + (range->stop - previous) + 1 > buffer.used - previous + ? buffer.used - previous + : (range->stop - previous) + 1 + ); f_fss_fail_utf_to_false(state); + if (F_status_is_error(state->status)) return; if (state->status == F_true) { @@ -529,10 +570,16 @@ extern "C" { for (;;) { - width_max = (range->stop - range->start) + 1; + if (range->start >= buffer.used) { + state->status = F_okay_eos; - if (width_max > buffer.used - range->start) { - width_max = buffer.used - range->start; + return; + } + + if (range->start > range->stop) { + state->status = F_okay_stop; + + return; } if (buffer.string[range->start] == f_fss_eol_s.string[0]) { @@ -547,35 +594,54 @@ extern "C" { continue; } - state->status = f_utf_is_whitespace(buffer.string + range->start, width_max, F_false); + state->status = f_utf_is_whitespace( + buffer.string + range->start, + (range->stop - range->start) + 1 > buffer.used - range->start + ? buffer.used - range->start + : (range->stop - range->start) + 1, + F_false + ); f_fss_fail_utf_to_false(state); if (state->status == F_false) { - state->status = f_utf_is_control(buffer.string + range->start, width_max); + state->status = f_utf_is_combining_joiner( + buffer.string + range->start, + (range->stop - range->start) + 1 > buffer.used - range->start + ? buffer.used - range->start + : (range->stop - range->start) + 1 + ); f_fss_fail_utf_to_false(state); + // This is combining, so any previous white space is no longer considered white space as such. + if (state->status == F_true) { + range->start = previous; + + break; + } + if (state->status == F_false) { - state->status = f_utf_is_combining(buffer.string + range->start, width_max); + state->status = f_utf_is_control( + buffer.string + range->start, + (range->stop - range->start) + 1 > buffer.used - range->start + ? buffer.used - range->start + : (range->stop - range->start) + 1 + ); f_fss_fail_utf_to_false(state); if (state->status == F_false) { - state->status = f_utf_is_zero_width(buffer.string + range->start, width_max); + state->status = f_utf_is_zero_width( + buffer.string + range->start, + (range->stop - range->start) + 1 > buffer.used - range->start + ? buffer.used - range->start + : (range->stop - range->start) + 1 + ); f_fss_fail_utf_to_false(state); if (state->status == F_false) break; } - else { - // This is combining, so any previous white space is no longer considered white space as such. - range->start = previous; - - // Ensure previous is not a place holder. - while (range->start && buffer.string[range->start] == f_fss_placeholder_s.string[0]) --range->start; - - break; - } } } @@ -583,10 +649,8 @@ extern "C" { previous = range->start; - width = macro_f_utf_byte_width(buffer.string[range->start]); - - if (width > 1) { - if (range->start + width >= buffer.used) { + if (macro_f_utf_byte_width_is(buffer.string[previous])) { + if (previous + macro_f_utf_byte_width(buffer.string[previous]) >= buffer.used) { state->status = F_status_set_error(F_complete_not_utf_eos); f_fss_fail_utf(state); @@ -594,7 +658,7 @@ extern "C" { return; } - if (range->start + width > range->stop) { + if (previous + macro_f_utf_byte_width(buffer.string[previous]) > range->stop) { state->status = F_status_set_error(F_complete_not_utf_stop); f_fss_fail_utf(state); @@ -603,19 +667,7 @@ extern "C" { } } - range->start += width; - - if (range->start >= buffer.used) { - state->status = F_okay_eos; - - return; - } - - if (range->start > range->stop) { - state->status = F_okay_stop; - - return; - } + range->start += macro_f_utf_byte_width(buffer.string[previous]); } // for state->status = F_okay; diff --git a/level_0/f_fss/c/fss.h b/level_0/f_fss/c/fss.h index b668ad01b..2bd98e299 100644 --- a/level_0/f_fss/c/fss.h +++ b/level_0/f_fss/c/fss.h @@ -200,17 +200,18 @@ extern "C" { * A state for providing flags and handling interrupts during long running operations. * * This alters state.status: - * F_true if the character in the buffer is a combining character. + * F_data_not on success but nothing is processed. * F_false if the character in the buffer is not a combining character. + * F_true if the character in the buffer is a combining character. * * F_parameter (with error bit) if a parameter is invalid. * * Errors (with error bit) from: f_utf_is_combining(). * * @return - * F_true if the character in the buffer is a combining character. * F_false if the character in the buffer is not a combining character. * F_false is returned on any error. + * F_true if the character in the buffer is a combining character. * * @see f_utf_is_combining() */ @@ -221,6 +222,8 @@ extern "C" { /** * Identify whether or not a character in the buffer is a graph (ASCII or UTF-8) character. * + * This does not check characters beyond the current one to check for combining or joining characters. + * * @param buffer * The string to process. * @param range @@ -229,17 +232,18 @@ extern "C" { * A state for providing flags and handling interrupts during long running operations. * * This alters state.status: - * F_true if the character in the buffer is a graph character. + * F_data_not on success but nothing is processed. * F_false if the character in the buffer is not a graph character. + * F_true on success and is a graph character. * * F_parameter (with error bit) if a parameter is invalid. * * Errors (with error bit) from: f_utf_is_graph(). * * @return - * F_true if the character in the buffer is a graph character. * F_false if the character in the buffer is not a graph character. * F_false is returned on any error. + * F_true if the character in the buffer is a graph character. * * @see f_utf_is_graph() */ @@ -248,7 +252,9 @@ extern "C" { #endif // _di_f_fss_is_graph_ /** - * Identify whether or not a character in the buffer is a non-zero-width whitespace or non-zero-width control (ASCII or UTF-8) character. + * Identify whether or not a character in the buffer is a non-zero-width white space or non-zero-width control (ASCII or UTF-8) character. + * + * This does not check characters beyond the current one to check for combining or joining characters. * * @param buffer * The string to process. @@ -258,8 +264,12 @@ extern "C" { * A state for providing flags and handling interrupts during long running operations. * * This alters state.status: - * F_okay on success. * F_data_not on success but nothing is processed. + * F_false if the character in the buffer is not a space character. + * F_space if the character is a zero width space (F_false is returned by this function). + * F_control if the character is a control character (F_false is returned by this function when zero-width or F_true when is both a space and a control character). + * F_true on success and is a space character. + * * F_parameter (with error bit) if a parameter is invalid. * * Errors (with error bit) from: f_utf_is_control(). @@ -267,9 +277,9 @@ extern "C" { * Errors (with error bit) from: f_utf_is_zero_width(). * * @return - * F_true if the character in the buffer is a space character. * F_false if the character in the buffer is not a space character. * F_false is returned on any error. + * F_true if the character in the buffer is a space character. * * @see f_utf_is_control() * @see f_utf_is_whitespace() @@ -282,6 +292,8 @@ extern "C" { /** * Identify whether or not a character in the buffer is a zero-width (ASCII or UTF-8) character. * + * This does not check characters beyond the current one to check for combining or joining characters. + * * The NULL character (U+0000) is a zero-width character. * * @param buffer @@ -294,17 +306,18 @@ extern "C" { * A state for providing flags and handling interrupts during long running operations. * * This alters state.status: - * F_true if the character in the buffer is a zero-width character. + * F_data_not on success but nothing is processed. * F_false if the character in the buffer is not a zero-width character. + * F_true if the character in the buffer is a zero-width character. * * F_parameter (with error bit) if a parameter is invalid. * * Errors (with error bit) from: f_utf_is_zero_width(). * * @return - * F_true if the character in the buffer is a zero-width character. * F_false if the character in the buffer is not a zero-width character. * F_false is returned on any error. + * F_true if the character in the buffer is a zero-width character. * * @see f_utf_is_zero_width() */ @@ -341,9 +354,9 @@ extern "C" { #endif // _di_f_fss_seek_to_eol_ /** - * Skip past all combining characters. + * Skip past all combining and joiner characters. * - * This will do nothing if the first character is not a combining character. + * This will do nothing if the first character is not a combining character and is not a joiner character. * * This skips past place holders. * @@ -360,7 +373,6 @@ extern "C" { * This alters state.status: * F_okay on success. * F_data_not on success but buffer.used is 0, initial range.start is greater than range.stop, or initial range.start is greater than or equal to buffer.used. - * F_is_not on success, but the first character is not a combining character. * F_okay_eol on success and EOL was reached. * F_okay_eos on success and EOS was reached. * F_okay_stop on success and stop point was reached. @@ -370,9 +382,9 @@ extern "C" { * F_complete_not_utf_stop (with error bit) if unable to get entire UTF-8 sequence due to stop point reached. * F_parameter (with error bit) if a parameter is invalid. * - * Errors (with error bit) from: f_utf_is_combining(). + * Errors (with error bit) from: f_utf_is_combining_joiner(). * - * @see f_utf_is_combining() + * @see f_utf_is_combining_joiner() */ #ifndef _di_f_fss_skip_past_combining_ extern void f_fss_skip_past_combining(const f_string_static_t buffer, f_range_t * const range, f_state_t * const state); @@ -406,6 +418,7 @@ extern "C" { * * If the first character in the given range is a combining character, then this will not skip past anything. * This is because combining characters apply from right to left. + * The caller should check the return status and handle accordingly in this situation. * * White space is checked for combining characters after the white space. * If there is a combining character after, then the white space is not consider as such. @@ -430,16 +443,16 @@ extern "C" { * F_okay_stop on success and stop point was reached. * * F_complete_not_utf_eos (with error bit) if unable to get entire UTF-8 sequence due to EOS. - * F_complete_not_utf_start (with error bit) if the first character is a combining character. + * F_complete_not_utf_start (with error bit) if the first character is a combining or a joiner character. * F_complete_not_utf_stop (with error bit) if unable to get entire UTF-8 sequence due to stop point reached. * F_parameter (with error bit) if a parameter is invalid. * - * Errors (with error bit) from: f_utf_is_combining(). + * Errors (with error bit) from: f_utf_is_combining_joiner(). * Errors (with error bit) from: f_utf_is_control(). * Errors (with error bit) from: f_utf_is_whitespace(). * Errors (with error bit) from: f_utf_is_zero_width(). * - * @see f_utf_is_combining() + * @see f_utf_is_combining_joiner() * @see f_utf_is_control() * @see f_utf_is_whitespace() * @see f_utf_is_zero_width() diff --git a/level_0/f_fss/data/build/settings-tests b/level_0/f_fss/data/build/settings-tests index 0b6b643b8..95d91a853 100644 --- a/level_0/f_fss/data/build/settings-tests +++ b/level_0/f_fss/data/build/settings-tests @@ -25,7 +25,7 @@ build_language c build_libraries -lc -lcmocka build_libraries-individual -lf_memory -lf_string -lf_type_array -lf_utf -lf_fss -build_sources_program test-fss-apply_delimit.c test-fss-apply_delimit_range.c test-fss-count_lines.c test-fss-count_lines_range.c test-fss-fail_utf.c test-fss-fail_utf_to_false.c test-fss-is_combining.c test-fss-is_graph.c test-fss-is_space.c test-fss-is_zero_width.c test-fss-seek_to_eol.c test-fss-skip_past_delimit.c test-fss-skip_past_space.c +build_sources_program test-fss-apply_delimit.c test-fss-apply_delimit_range.c test-fss-count_lines.c test-fss-count_lines_range.c test-fss-fail_utf.c test-fss-fail_utf_to_false.c test-fss-is_combining.c test-fss-is_graph.c test-fss-is_space.c test-fss-is_zero_width.c test-fss-seek_to_eol.c test-fss-skip_past_combining.c test-fss-skip_past_delimit.c test-fss-skip_past_space.c build_sources_program test-fss-item_delete.c test-fss-item_destroy.c build_sources_program test-fss-items_delete_callback.c test-fss-items_destroy_callback.c test-fss-itemss_delete_callback.c test-fss-itemss_destroy_callback.c build_sources_program test-fss-named_delete.c test-fss-named_destroy.c diff --git a/level_0/f_fss/data/documentation/man/man3/f_fss_is_graph.3 b/level_0/f_fss/data/documentation/man/man3/f_fss_is_graph.3 index 92d7765e6..594c63102 100644 --- a/level_0/f_fss/data/documentation/man/man3/f_fss_is_graph.3 +++ b/level_0/f_fss/data/documentation/man/man3/f_fss_is_graph.3 @@ -15,7 +15,9 @@ f_fss_is_graph \- .PP Identify whether or not a character in the buffer is a graph (ASCII or UTF-8) character. .PP -This alters state.status: F_true if the character in the buffer is a graph character. F_false if the character in the buffer is not a graph character. +If the character is not a graph character but is followed by a combining character, then it is considered a graph character due to the combination result. +.PP +This alters state.status: F_true if the character in the buffer is a graph character or the character is followed by a combining character. F_false if the character in the buffer is not a graph character. .PP F_parameter (with error bit) if a parameter is invalid. .PP @@ -81,7 +83,7 @@ struct f_range_t { .RE .SH RETURN VALUE .PP -F_true if the character in the buffer is a graph character. F_false if the character in the buffer is not a graph character. F_false is returned on any error. +F_true if the character in the buffer is a graph character or the character is followed by a combining character. F_false if the character in the buffer is not a graph character. F_false is returned on any error. .SH SEE ALSO .PP .nh diff --git a/level_0/f_fss/data/documentation/man/man3/f_fss_is_space.3 b/level_0/f_fss/data/documentation/man/man3/f_fss_is_space.3 index 9f8a4865a..c5dde9ade 100644 --- a/level_0/f_fss/data/documentation/man/man3/f_fss_is_space.3 +++ b/level_0/f_fss/data/documentation/man/man3/f_fss_is_space.3 @@ -15,6 +15,8 @@ f_fss_is_space \- .PP Identify whether or not a character in the buffer is a non-zero-width whitespace or non-zero-width control (ASCII or UTF-8) character. .PP +A valid white space (or control character) that is followed by a combining character is considered a non-space unless that control character is a new line. The new line is an exception due to the current handling of that character with combining characters as observed in common editors. +.PP This alters state.status: F_okay on success. F_data_not on success but nothing is processed. F_parameter (with error bit) if a parameter is invalid. .PP Errors (with error bit) from: f_utf_is_control(). Errors (with error bit) from: f_utf_is_whitespace(). Errors (with error bit) from: f_utf_is_zero_width(). @@ -79,7 +81,7 @@ struct f_range_t { .RE .SH RETURN VALUE .PP -F_true if the character in the buffer is a space character. F_false if the character in the buffer is not a space character. F_false is returned on any error. +F_true if the character in the buffer is a space character (and the following character, if any, is not a combining character). F_false if the character in the buffer is not a space character. F_false is returned on any error. .SH SEE ALSO .PP .nh diff --git a/level_0/f_fss/tests/unit/c/test-fss-is_space.c b/level_0/f_fss/tests/unit/c/test-fss-is_space.c index 546d97455..aadd8db3e 100644 --- a/level_0/f_fss/tests/unit/c/test-fss-is_space.c +++ b/level_0/f_fss/tests/unit/c/test-fss-is_space.c @@ -40,8 +40,8 @@ void test__f_fss_is_space__works(void **state) { F_false, F_true, F_false, - F_false, - F_false, + F_true, + F_true, F_false, }; @@ -52,7 +52,7 @@ void test__f_fss_is_space__works(void **state) { const f_status_t status = f_fss_is_space(tests[i], range, &state_data); if (status != expects[i]) { - printf("[ -------> ] --- [00] Expected %s at index %d.\n", expects[i] ? "false" : "true", i); + printf("[ -------> ] --- [00] Expected %s at index %d for string '%s'.\n", expects[i] ? "true" : "false", i, tests[i].used ? tests[i].string : ""); } assert_int_equal(status, expects[i]); @@ -64,10 +64,16 @@ void test__f_fss_is_space__works(void **state) { const f_status_t status = f_fss_is_space(f_string_empty_s, range, &state_data); if (status) { - printf("[ -------> ] --- [01] Expected false for empty string with range (1, 0).\n"); + printf("[ -------> ] --- [01] Expected true for empty string with range (%lu, %lu).\n", range.start, range.stop); } assert_int_equal(status, F_false); + + if (state_data.status != F_data_not) { + printf("[ -------> ] --- [02] Expected F_data_not in state.status for empty string with range (%lu, %lu).\n", range.start, range.stop); + } + + assert_int_equal(state_data.status, F_data_not); } { @@ -76,10 +82,16 @@ void test__f_fss_is_space__works(void **state) { const f_status_t status = f_fss_is_space(f_string_empty_s, range, &state_data); if (status) { - printf("[ -------> ] --- [01] Expected false for empty string with (out of) range (%lu, %lu).\n", range.start, range.stop); + printf("[ -------> ] --- [03] Expected true for empty string with (out of) range (%lu, %lu).\n", range.start, range.stop); } assert_int_equal(status, F_false); + + if (state_data.status != F_data_not) { + printf("[ -------> ] --- [04] Expected F_data_not in state.status for empty string with (out of) range (%lu, %lu).\n", range.start, range.stop); + } + + assert_int_equal(state_data.status, F_data_not); } } diff --git a/level_0/f_fss/tests/unit/c/test-fss-skip_past_combining.c b/level_0/f_fss/tests/unit/c/test-fss-skip_past_combining.c new file mode 100644 index 000000000..e503c41e5 --- /dev/null +++ b/level_0/f_fss/tests/unit/c/test-fss-skip_past_combining.c @@ -0,0 +1,183 @@ +#include "test-fss.h" +#include "test-fss-skip_past_combining.h" + +#ifdef __cplusplus +extern "C" { +#endif + +void test__f_fss_skip_past_combining__parameter_checking(void **state) { + + const f_string_static_t test = macro_f_string_static_t_initialize_1("test", 4, 0); + f_state_t state_data = f_state_t_initialize; + + { + f_fss_skip_past_combining(test, 0, &state_data); + + assert_int_equal(state_data.status, F_status_set_error(F_parameter)); + } +} + +void test__f_fss_skip_past_combining__returns_data_not(void **state) { + + const f_string_static_t test = macro_f_string_static_t_initialize_1("test", 4, 0); + f_state_t state_data = f_state_t_initialize; + + { + f_range_t range = f_range_t_initialize; + + f_fss_skip_past_combining(f_string_empty_s, &range, &state_data); + + assert_int_equal(state_data.status, F_data_not); + } + + { + f_range_t range = f_range_t_initialize; + + f_fss_skip_past_combining(test, &range, &state_data); + + assert_int_equal(state_data.status, F_data_not); + } + + { + f_range_t range = macro_f_range_t_initialize_1(0, test.used - 1); + + f_fss_skip_past_combining(f_string_empty_s, &range, &state_data); + + assert_int_equal(state_data.status, F_data_not); + } + + { + f_range_t range = macro_f_range_t_initialize_1(test.used, test.used + 2); + + f_fss_skip_past_combining(test, &range, &state_data); + + assert_int_equal(state_data.status, F_data_not); + } +} + +void test__f_fss_skip_past_combining__works(void **state) { + + f_state_t state_data = f_state_t_initialize; + + f_string_static_t tests[] = { + + // Set 1 + macro_f_string_static_t_initialize_1("́", 2, 0), + macro_f_string_static_t_initialize_1("\0́", 3, 0), + macro_f_string_static_t_initialize_1("\0\0\0́", 5, 0), + macro_f_string_static_t_initialize_1("\0\0\0́\0\0", 7, 0), + + // Set 2 + macro_f_string_static_t_initialize_1("́.", 3, 0), + macro_f_string_static_t_initialize_1("\0̃.", 4, 0), + macro_f_string_static_t_initialize_1("\0\0\0̃.", 6, 0), + macro_f_string_static_t_initialize_1("\0\0\0̃\0\0.", 8, 0), + + // Set 3 + macro_f_string_static_t_initialize_1("̠.", 3, 0), + macro_f_string_static_t_initialize_1("\0̠.", 4, 0), + macro_f_string_static_t_initialize_1("\0\0\0̠.", 6, 0), + macro_f_string_static_t_initialize_1("\0\0\0̠\0\0.", 8, 0), + + // Set 4 + macro_f_string_static_t_initialize_1("̠́.", 5, 0), + macro_f_string_static_t_initialize_1("\0̠́.", 6, 0), + macro_f_string_static_t_initialize_1("\0\0\0̠́.", 8, 0), + macro_f_string_static_t_initialize_1("\0\0\0̠́\0\0.", 10, 0), + + // Set 5 + macro_f_string_static_t_initialize_1("̠́́.", 7, 0), + macro_f_string_static_t_initialize_1("\0̠́́.", 8, 0), + macro_f_string_static_t_initialize_1("\0\0\0̠́́.", 10, 0), + macro_f_string_static_t_initialize_1("\0\0\0̠́́\0\0.", 12, 0), + }; + + uint16_t statuss[] = { + + // Set 1 + F_okay_eos, + F_okay_eos, + F_okay_eos, + F_okay_eos, + + // Set 2 + F_okay, + F_okay, + F_okay, + F_okay, + + // Set 3 + F_okay, + F_okay, + F_okay, + F_okay, + + // Set 4 + F_okay, + F_okay, + F_okay, + F_okay, + + // Set 5 + F_okay, + F_okay, + F_okay, + F_okay, + }; + + f_number_unsigned_t ranges[] = { + + // Set 1 + 2, + 3, + 5, + 7, + + // Set 2 + 2, + 3, + 5, + 7, + + // Set 3 + 2, + 3, + 5, + 7, + + // Set 4 + 4, + 5, + 7, + 9, + + // Set 5 + 6, + 7, + 9, + 11, + }; + + for (uint8_t i = 0; i < 20; ++i) { + + f_range_t range = macro_f_range_t_initialize_1(0, tests[i].used - 1); + + f_fss_skip_past_combining(tests[i], &range, &state_data); + + if (state_data.status != statuss[i]) { + printf("[ -------> ] --- [00] Expected %u in state.status but got %u at index %d for string '%s'.\n", statuss[i], state_data.status, i, tests[i].used ? tests[i].string : ""); + } + + assert_int_equal(state_data.status, statuss[i]); + + if (range.start != ranges[i]) { + printf("[ -------> ] --- [01] Expected %lu in range.start but got %lu at index %d for string '%s'.\n", ranges[i], range.start, i, tests[i].used ? tests[i].string : ""); + } + + assert_int_equal(range.start, ranges[i]); + } // for +} + +#ifdef __cplusplus +} // extern "C" +#endif diff --git a/level_0/f_fss/tests/unit/c/test-fss-skip_past_combining.h b/level_0/f_fss/tests/unit/c/test-fss-skip_past_combining.h new file mode 100644 index 000000000..886fcfb9d --- /dev/null +++ b/level_0/f_fss/tests/unit/c/test-fss-skip_past_combining.h @@ -0,0 +1,34 @@ +/** + * FLL - Level 0 + * + * Project: FSS + * API Version: 0.8 + * Licenses: lgpl-2.1-or-later + * + * Test skip past combining in the fss project. + */ +#ifndef _TEST__F_fss__skip_past_combining +#define _TEST__F_fss__skip_past_combining + +/** + * Test that the function correctly fails on invalid parameter. + * + * @see f_fss_skip_past_combining() + */ +extern void test__f_fss_skip_past_combining__parameter_checking(void **state); + +/** + * Test that the function returns F_data_not. + * + * @see f_fss_skip_past_combining() + */ +extern void test__f_fss_skip_past_combining__returns_data_not(void **state); + +/** + * Test that the function works. + * + * @see f_fss_skip_past_combining() + */ +extern void test__f_fss_skip_past_combining__works(void **state); + +#endif // _TEST__F_fss__skip_past_combining diff --git a/level_0/f_fss/tests/unit/c/test-fss-skip_past_delimit.c b/level_0/f_fss/tests/unit/c/test-fss-skip_past_delimit.c index 121189d2e..790aef579 100644 --- a/level_0/f_fss/tests/unit/c/test-fss-skip_past_delimit.c +++ b/level_0/f_fss/tests/unit/c/test-fss-skip_past_delimit.c @@ -66,7 +66,14 @@ void test__f_fss_skip_past_delimit__works(void **state) { macro_f_string_static_t_initialize_1("\0\0\0test\0\0", 9, 0), }; - f_number_unsigned_t expects[] = { + uint16_t statuss[] = { + F_okay, + F_okay, + F_okay, + F_okay, + }; + + f_number_unsigned_t ranges[] = { 0, 1, 3, @@ -79,8 +86,16 @@ void test__f_fss_skip_past_delimit__works(void **state) { f_fss_skip_past_delimit(tests[i], &range, &state_data); - assert_int_equal(state_data.status, F_okay); - assert_int_equal(range.start, expects[i]); + if (state_data.status != statuss[i]) { + printf("[ -------> ] --- [00] Expected %u in state.status but got %u at index %d for string '%s'.\n", statuss[i], state_data.status, i, tests[i].used ? tests[i].string : ""); + } + + assert_int_equal(state_data.status, statuss[i]); + + if (range.start != ranges[i]) { + printf("[ -------> ] --- [01] Expected %lu in range.start but got %lu at index %d for string '%s'.\n", ranges[i], range.start, i, tests[i].used ? tests[i].string : ""); + } + } // for } diff --git a/level_0/f_fss/tests/unit/c/test-fss-skip_past_space.c b/level_0/f_fss/tests/unit/c/test-fss-skip_past_space.c index e61026ed4..95ddd01cc 100644 --- a/level_0/f_fss/tests/unit/c/test-fss-skip_past_space.c +++ b/level_0/f_fss/tests/unit/c/test-fss-skip_past_space.c @@ -59,34 +59,77 @@ void test__f_fss_skip_past_space__works(void **state) { f_state_t state_data = f_state_t_initialize; + // Space followed by a zero-width joiner and a joined character with NULLs before joiner (U+0020 \0 \0 U+200D U+1F9D4). + char space_null_join[] = { 0x20, 0x0, 0x0, 0xe2, 0x80, 0x8d, 0xf0, 0x9f, 0xa7, 0x94 }; + f_string_static_t tests[] = { - macro_f_string_static_t_initialize_1("test", 4, 0), - macro_f_string_static_t_initialize_1(" test", 5, 0), - macro_f_string_static_t_initialize_1(" test", 7, 0), - macro_f_string_static_t_initialize_1(" \0test ", 9, 0), - macro_f_string_static_t_initialize_1(" \t\atest ", 9, 0), - macro_f_string_static_t_initialize_1("\a\t $test ", 10, 0), - macro_f_string_static_t_initialize_1(" ́test", 7, 0), + macro_f_string_static_t_initialize_1("test", 4, 0), + macro_f_string_static_t_initialize_1(" test", 5, 0), + macro_f_string_static_t_initialize_1(" test", 7, 0), + macro_f_string_static_t_initialize_1(" \0test ", 9, 0), + macro_f_string_static_t_initialize_1(" \t\atest ", 9, 0), + macro_f_string_static_t_initialize_1("\a\t $test ", 10, 0), + macro_f_string_static_t_initialize_1(" ́test", 7, 0), // Space followed by a combining character (U+0020 U+0301). + macro_f_string_static_t_initialize_1("​test", 7, 0), // Starts with zero-width space (U+200B). + macro_f_string_static_t_initialize_1(" ‍🧔test", 12, 0), // Space followed by a zero-width joiner and a joined character (U+0020 U+200D U+1F9D4). + macro_f_string_static_t_initialize_1(" ⁠test", 8, 0), // Space followed by a word joiner and a joined character (U+0020 U+2060 U+0062). + macro_f_string_static_t_initialize_1("́⁠test", 6, 0), // Starts with combining character (U+0301). + macro_f_string_static_t_initialize_1("‍́test", 6, 0), // Starts with joiner character (U+2060). + macro_f_string_static_t_initialize_1(" ‍\0\0🧔test", 14, 0), // Space followed by a zero-width joiner and a joined character with NULLs after joiner (U+0020 U+200D \0 \0 U+1F9D4). + macro_f_string_static_t_initialize_1(space_null_join, 14, 0), + }; + + uint16_t statuss[] = { + F_okay, + F_okay, + F_okay, + F_okay, + F_okay, + F_okay, + F_okay, + F_okay, + F_okay, + F_okay, + F_status_set_error(F_complete_not_utf_start), + F_status_set_error(F_complete_not_utf_start), + F_okay, + F_okay, }; - f_number_unsigned_t expects[] = { + f_number_unsigned_t ranges[] = { 0, 1, 3, 3, 3, 3, + 0, 3, + 0, + 0, + 0, + 0, + 0, + 0, }; - for (uint8_t i = 0; i < 7; ++i) { + for (uint8_t i = 0; i < 14; ++i) { f_range_t range = macro_f_range_t_initialize_1(0, tests[i].used - 1); f_fss_skip_past_space(tests[i], &range, &state_data); - assert_int_equal(state_data.status, F_okay); - assert_int_equal(range.start, expects[i]); + if (state_data.status != statuss[i]) { + printf("[ -------> ] --- [00] Expected %u in state.status but got %u at index %d for string '%s'.\n", statuss[i], state_data.status, i, tests[i].used ? tests[i].string : ""); + } + + assert_int_equal(state_data.status, statuss[i]); + + if (range.start != ranges[i]) { + printf("[ -------> ] --- [01] Expected %lu in range.start but got %lu at index %d for string '%s'.\n", ranges[i], range.start, i, tests[i].used ? tests[i].string : ""); + } + + assert_int_equal(range.start, ranges[i]); } // for } diff --git a/level_0/f_fss/tests/unit/c/test-fss.c b/level_0/f_fss/tests/unit/c/test-fss.c index 999dbb08b..399fe451f 100644 --- a/level_0/f_fss/tests/unit/c/test-fss.c +++ b/level_0/f_fss/tests/unit/c/test-fss.c @@ -24,6 +24,7 @@ int main(void) { cmocka_unit_test(test__f_fss_count_lines__returns_data_not), cmocka_unit_test(test__f_fss_count_lines_range__returns_data_not), cmocka_unit_test(test__f_fss_seek_to_eol__returns_data_not), + cmocka_unit_test(test__f_fss_skip_past_combining__returns_data_not), cmocka_unit_test(test__f_fss_skip_past_delimit__returns_data_not), cmocka_unit_test(test__f_fss_skip_past_space__returns_data_not), @@ -36,6 +37,7 @@ int main(void) { cmocka_unit_test(test__f_fss_is_space__works), cmocka_unit_test(test__f_fss_is_zero_width__works), cmocka_unit_test(test__f_fss_seek_to_eol__works), + cmocka_unit_test(test__f_fss_skip_past_combining__works), cmocka_unit_test(test__f_fss_skip_past_delimit__works), cmocka_unit_test(test__f_fss_skip_past_space__works), @@ -152,6 +154,7 @@ int main(void) { // f_fss_is_space() doesn't use parameter checking. // f_fss_is_zero_width() doesn't use parameter checking. cmocka_unit_test(test__f_fss_seek_to_eol__parameter_checking), + cmocka_unit_test(test__f_fss_skip_past_combining__parameter_checking), cmocka_unit_test(test__f_fss_skip_past_delimit__parameter_checking), cmocka_unit_test(test__f_fss_skip_past_space__parameter_checking), diff --git a/level_0/f_fss/tests/unit/c/test-fss.h b/level_0/f_fss/tests/unit/c/test-fss.h index 043d15954..964c18ede 100644 --- a/level_0/f_fss/tests/unit/c/test-fss.h +++ b/level_0/f_fss/tests/unit/c/test-fss.h @@ -76,6 +76,7 @@ #include "test-fss-simple_packets_destroy_callback.h" #include "test-fss-simple_packetss_delete_callback.h" #include "test-fss-simple_packetss_destroy_callback.h" +#include "test-fss-skip_past_combining.h" #include "test-fss-skip_past_delimit.h" #include "test-fss-skip_past_space.h" diff --git a/level_0/f_parse/c/parse.c b/level_0/f_parse/c/parse.c index 94f9a8cb5..92857d73f 100644 --- a/level_0/f_parse/c/parse.c +++ b/level_0/f_parse/c/parse.c @@ -38,7 +38,7 @@ extern "C" { } else { if (range->start + width > range->stop) return F_status_set_error(F_complete_not_utf_stop); - if (range->start + width >= buffer.used) return F_status_set_error(F_complete_not_utf_eos); + if (range->start + width > buffer.used) return F_status_set_error(F_complete_not_utf_eos); if (width == seek_width) { f_utf_char_t character = 0; @@ -91,7 +91,7 @@ extern "C" { } else { if (range->start + width > range->stop) return F_status_set_error(F_complete_not_utf_stop); - if (range->start + width >= buffer.used) return F_status_set_error(F_complete_not_utf_eos); + if (range->start + width > buffer.used) return F_status_set_error(F_complete_not_utf_eos); } range->start += width; @@ -140,7 +140,7 @@ extern "C" { } else { if (range->start + width > range->stop) return F_status_set_error(F_complete_not_utf_stop); - if (range->start + width >= buffer.used) return F_status_set_error(F_complete_not_utf_eos); + if (range->start + width > buffer.used) return F_status_set_error(F_complete_not_utf_eos); } range->start += width; @@ -194,7 +194,7 @@ extern "C" { } else { if (range->start + width > range->stop) return F_status_set_error(F_complete_not_utf_stop); - if (range->start + width >= buffer.used) return F_status_set_error(F_complete_not_utf_eos); + if (range->start + width > buffer.used) return F_status_set_error(F_complete_not_utf_eos); if (width == seek_width) { f_utf_char_t character = 0; diff --git a/level_0/f_utf/c/private-utf_combining.c b/level_0/f_utf/c/private-utf_combining.c index dd30ea25b..e9f73f591 100644 --- a/level_0/f_utf/c/private-utf_combining.c +++ b/level_0/f_utf/c/private-utf_combining.c @@ -6,7 +6,7 @@ extern "C" { #endif -#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_is_combining_) +#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_is_combining_) || !defined(_di_f_utf_is_combining_joiner_) f_status_t private_f_utf_character_is_combining(const f_utf_char_t sequence) { if (macro_f_utf_char_t_width_is(sequence) == 2) { @@ -1485,7 +1485,7 @@ extern "C" { return F_false; } -#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_is_combining_) +#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_is_combining_) || !defined(_di_f_utf_is_combining_joiner_) #ifdef __cplusplus } // extern "C" diff --git a/level_0/f_utf/c/private-utf_combining.h b/level_0/f_utf/c/private-utf_combining.h index a20b24c08..4a7c6327d 100644 --- a/level_0/f_utf/c/private-utf_combining.h +++ b/level_0/f_utf/c/private-utf_combining.h @@ -36,11 +36,13 @@ extern "C" { * * @see f_utf_character_is_alphabetic() * @see f_utf_character_is_combining() + * @see f_utf_character_is_combining_joiner() * @see f_utf_is_combining() + * @see f_utf_is_combining_joiner() */ -#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_is_combining_) +#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_is_combining_) || !defined(_di_f_utf_is_combining_joiner_) extern f_status_t private_f_utf_character_is_combining(const f_utf_char_t sequence) F_attribute_visibility_internal_d; -#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_is_combining_) +#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_character_is_combining_joiner_)|| !defined(_di_f_utf_is_combining_) || !defined(_di_f_utf_is_combining_joiner_) #ifdef __cplusplus } // extern "C" diff --git a/level_0/f_utf/c/private-utf_joiner.c b/level_0/f_utf/c/private-utf_joiner.c index c5820ea33..c525c28ff 100644 --- a/level_0/f_utf/c/private-utf_joiner.c +++ b/level_0/f_utf/c/private-utf_joiner.c @@ -6,7 +6,7 @@ extern "C" { #endif -#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_) +#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_combining_joiner_) || !defined(_di_f_utf_is_joiner_) f_status_t private_f_utf_character_is_joiner(const f_utf_char_t sequence) { if (macro_f_utf_char_t_width_is(sequence) == 3) { @@ -22,7 +22,7 @@ extern "C" { return F_false; } -#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_) +#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_combining_joiner_) || !defined(_di_f_utf_is_joiner_) #ifdef __cplusplus } // extern "C" diff --git a/level_0/f_utf/c/private-utf_joiner.h b/level_0/f_utf/c/private-utf_joiner.h index f61aaf7e2..c80f10c85 100644 --- a/level_0/f_utf/c/private-utf_joiner.h +++ b/level_0/f_utf/c/private-utf_joiner.h @@ -37,15 +37,17 @@ extern "C" { * @see f_utf_character_is_alphabetic() * @see f_utf_character_is_alphabetic_digit() * @see f_utf_character_is_alphabetic_numeric() + * @see f_utf_character_is_combining_joiner() * @see f_utf_character_is_joiner() * @see f_utf_is_alphabetic() * @see f_utf_is_alphabetic_digit() * @see f_utf_is_alphabetic_numeric() + * @see f_utf_is_combining_joiner() * @see f_utf_is_joiner() */ -#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_) +#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_) extern f_status_t private_f_utf_character_is_joiner(const f_utf_char_t sequence) F_attribute_visibility_internal_d; -#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_) +#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_) #ifdef __cplusplus } // extern "C" diff --git a/level_0/f_utf/c/private-utf_zero_width.c b/level_0/f_utf/c/private-utf_zero_width.c index d7f15f388..52e438fb6 100644 --- a/level_0/f_utf/c/private-utf_zero_width.c +++ b/level_0/f_utf/c/private-utf_zero_width.c @@ -12,11 +12,21 @@ extern "C" { if (macro_f_utf_char_t_width_is(sequence) == 3) { if (macro_f_utf_char_t_to_char_1(sequence) == 0xe2) { + // General Punctuation: U+200D. + if (sequence == 0xe2808d00) { + return F_true; + } + // General Punctuation: U+2028 to U+2029. if (sequence >= 0xf090b4a3 && sequence <= 0xf090b4a7) { return F_true; } + // General Punctuation: U+2060. + if (sequence == 0xe281a000) { + return F_true; + } + // General Punctuation: U+2066 to U+2069. if (sequence >= 0xe281a600 && sequence <= 0xe281a900) { return F_true; diff --git a/level_0/f_utf/c/utf.c b/level_0/f_utf/c/utf.c index 5c04b1b8b..558c493a8 100644 --- a/level_0/f_utf/c/utf.c +++ b/level_0/f_utf/c/utf.c @@ -67,7 +67,7 @@ extern "C" { return F_okay_stop; } else if (range->start + width >= buffer.used) { - if (width > 1 && range->start + width >= buffer.used + 1) { + if (width > 1 && range->start + width > buffer.used) { return F_status_set_error(F_complete_not_utf_eos); } diff --git a/level_0/f_utf/c/utf/is.c b/level_0/f_utf/c/utf/is.c index 6137cc4ce..9e6305d6a 100644 --- a/level_0/f_utf/c/utf/is.c +++ b/level_0/f_utf/c/utf/is.c @@ -148,6 +148,32 @@ extern "C" { } #endif // _di_f_utf_is_combining_ +#ifndef _di_f_utf_is_combining_joiner_ + f_status_t f_utf_is_combining_joiner(const f_string_t sequence, const f_number_unsigned_t width_max) { + + if (width_max < 1) return F_data_not; + + if (macro_f_utf_byte_width_is(*sequence)) { + if (macro_f_utf_byte_width_is(*sequence) > width_max) return F_status_set_error(F_complete_not_utf); + if (macro_f_utf_byte_width_is(*sequence) == 1) return F_status_set_error(F_utf_fragment); + + f_utf_char_t utf = 0; + + { + const f_status_t status = private_f_utf_char_to_character(sequence, width_max, &utf); + if (F_status_is_error(status)) return status; + } + + if (private_f_utf_character_is_combining(utf)) return F_true; + + return private_f_utf_character_is_joiner(utf); + } + + // There are no ASCII combining characters. + return F_false; + } +#endif // _di_f_utf_is_combining_joiner_ + #ifndef _di_f_utf_is_control_ f_status_t f_utf_is_control(const f_string_t sequence, const f_number_unsigned_t width_max) { @@ -808,6 +834,7 @@ extern "C" { // Control and combining characters are zero-width. if (private_f_utf_character_is_control(utf)) return F_true; if (private_f_utf_character_is_combining(utf)) return F_true; + if (private_f_utf_character_is_joiner(utf)) return F_true; return private_f_utf_character_is_zero_width(utf); } diff --git a/level_0/f_utf/c/utf/is.h b/level_0/f_utf/c/utf/is.h index 805456ae1..2608477f1 100644 --- a/level_0/f_utf/c/utf/is.h +++ b/level_0/f_utf/c/utf/is.h @@ -163,6 +163,29 @@ extern "C" { extern f_status_t f_utf_is_combining(const f_string_t sequence, const f_number_unsigned_t width_max); #endif // _di_f_utf_is_combining_ +/** + * Check to see if the entire byte block of the character is a UTF-8 combining character or a UTF-8 joiner character. + * + * @param sequence + * The byte sequence to validate as a character. + * There must be enough space allocated to compare against, as limited by width_max. + * @param width_max + * The maximum width available for checking. + * Can be anything greater than 0. + * + * @return + * F_true if a UTF-8 combining character. + * F_false if not a UTF-8 combining character. + * F_data_not if width_max is less than 1. + * + * F_complete_not_utf (with error bit set) if character is an incomplete UTF-8 sequence. + * F_utf_fragment (with error bit) if character is a UTF-8 fragment. + * F_utf_not (with error bit) if Unicode is an invalid Unicode character. + */ +#ifndef _di_f_utf_is_combining_joiner_ + extern f_status_t f_utf_is_combining_joiner(const f_string_t sequence, const f_number_unsigned_t width_max); +#endif // _di_f_utf_is_combining_joiner_ + /** * Check to see if the entire byte block of the character is an ASCII or UTF-8 control character. * diff --git a/level_0/f_utf/c/utf/is_character.c b/level_0/f_utf/c/utf/is_character.c index 7ec0805e0..0786ebebc 100644 --- a/level_0/f_utf/c/utf/is_character.c +++ b/level_0/f_utf/c/utf/is_character.c @@ -107,6 +107,22 @@ extern "C" { } #endif // _di_f_utf_character_is_combining_ +#ifndef _di_f_utf_character_is_combining_joiner_ + f_status_t f_utf_character_is_combining_joiner(const f_utf_char_t sequence) { + + if (macro_f_utf_char_t_width_is(sequence)) { + if (macro_f_utf_char_t_width_is(sequence) == 1) return F_status_set_error(F_utf_fragment); + + if (private_f_utf_character_is_combining(sequence)) return F_true; + + return private_f_utf_character_is_joiner(sequence); + } + + // There are no combining characters in ASCII. + return F_false; + } +#endif // _di_f_utf_character_is_combining_joiner_ + #ifndef _di_f_utf_character_is_control_ f_status_t f_utf_character_is_control(const f_utf_char_t sequence) { diff --git a/level_0/f_utf/c/utf/is_character.h b/level_0/f_utf/c/utf/is_character.h index b9cee3cc9..e2ae0dc1e 100644 --- a/level_0/f_utf/c/utf/is_character.h +++ b/level_0/f_utf/c/utf/is_character.h @@ -135,6 +135,23 @@ extern "C" { extern f_status_t f_utf_character_is_combining(const f_utf_char_t sequence); #endif // _di_f_utf_character_is_combining_ +/** + * Check to see if the entire byte block of the character is a UTF-8 combining character or a UTF-8 joiner character. + * + * @param sequence + * The byte sequence to validate as a character. + * + * @return + * F_true if a UTF-8 combining character. + * F_false if not a UTF-8 combining character. + * + * F_utf_fragment (with error bit) if character is a UTF-8 fragment. + * F_utf_not (with error bit) if unicode is an invalid Unicode character. + */ +#ifndef _di_f_utf_character_is_combining_joiner_ + extern f_status_t f_utf_character_is_combining_joiner(const f_utf_char_t sequence); +#endif // _di_f_utf_character_is_combining_joiner_ + /** * Check to see if the entire byte block of the character is an ASCII or UTF-8 control character. * diff --git a/level_0/f_utf/data/build/settings-tests b/level_0/f_utf/data/build/settings-tests index f621ce4b1..2b9f73353 100644 --- a/level_0/f_utf/data/build/settings-tests +++ b/level_0/f_utf/data/build/settings-tests @@ -30,6 +30,7 @@ build_sources_program test-utf-character_is_alphabetic.c test-utf-is_alphabetic. build_sources_program test-utf-character_is_alphabetic_digit.c test-utf-is_alphabetic_digit.c build_sources_program test-utf-character_is_alphabetic_numeric.c test-utf-is_alphabetic_numeric.c build_sources_program test-utf-character_is_combining.c test-utf-is_combining.c +build_sources_program test-utf-character_is_combining_joiner.c test-utf-is_combining_joiner.c build_sources_program test-utf-character_is_control.c test-utf-is_control.c build_sources_program test-utf-character_is_control_picture.c test-utf-is_control_picture.c build_sources_program test-utf-character_is_digit.c test-utf-is_digit.c diff --git a/level_0/f_utf/tests/unit/c/data-utf.c b/level_0/f_utf/tests/unit/c/data-utf.c index aba4f55ed..6299056a4 100644 --- a/level_0/f_utf/tests/unit/c/data-utf.c +++ b/level_0/f_utf/tests/unit/c/data-utf.c @@ -14,6 +14,11 @@ FILE *data__bytesequence_file_open__combining(void) { return fopen("./data/tests/bytesequences/combining-all.txt", "r"); } +FILE *data__bytesequence_file_open__combining_joiner(void) { + + return fopen("./data/tests/bytesequences/combining_joiner-all.txt", "r"); +} + FILE *data__bytesequence_file_open__control(void) { return fopen("./data/tests/bytesequences/control-all.txt", "r"); diff --git a/level_0/f_utf/tests/unit/c/data-utf.h b/level_0/f_utf/tests/unit/c/data-utf.h index 171dfa8ef..21a5b8ac8 100644 --- a/level_0/f_utf/tests/unit/c/data-utf.h +++ b/level_0/f_utf/tests/unit/c/data-utf.h @@ -52,6 +52,21 @@ extern FILE *data__bytesequence_file_open__alphabetic(void); */ extern FILE *data__bytesequence_file_open__combining(void); +/** + * Open the "combining" or "joiner" bytesequence file. + * + * This assumes the following: + * - The file path is relative to the current working directory (tests are run from project root). + * - The file path is "data/tests/bytesequences/combining_joiner-all.txt". + * + * @return + * Non-zero on success. + * 0 on failure. + * + * @see fopen() + */ +extern FILE *data__bytesequence_file_open__combining_joiner(void); + /** * Open the "control" bytesequence file. * diff --git a/level_0/f_utf/tests/unit/c/test-utf-character_is_combining_joiner.c b/level_0/f_utf/tests/unit/c/test-utf-character_is_combining_joiner.c new file mode 100644 index 000000000..461886639 --- /dev/null +++ b/level_0/f_utf/tests/unit/c/test-utf-character_is_combining_joiner.c @@ -0,0 +1,66 @@ +#include "test-utf.h" +#include "test-utf-character_is_combining_joiner.h" + +#ifdef __cplusplus +extern "C" { +#endif + +void test__f_utf_character_is_combining_joiner__works(void **state) { + + { + FILE *file_combining = data__bytesequence_file_open__combining(); + FILE *file_joiner = data__bytesequence_file_open__joiner(); + + assert_non_null(file_combining); + assert_non_null(file_joiner); + + f_utf_char_t sequence = 0; + ssize_t bytes = 0; + long long code = 0; + + f_number_unsigned_t line = 0; + + do { + bytes = data__bytesequence_get_line(file_combining, &sequence, &code); + + if (bytes > 0) { + const f_status_t status = f_utf_character_is_combining_joiner(sequence); + + if (status != F_true) { + printf("[ -------> ] --- [0] Failure at line=%lu sequence='%0x' (%llu).\n", line + 1, sequence, code); + } + + assert_int_equal(status, F_true); + } + + ++line; + + } while (bytes > 0); + + line = 0; + + do { + bytes = data__bytesequence_get_line(file_joiner, &sequence, &code); + + if (bytes > 0) { + const f_status_t status = f_utf_character_is_combining_joiner(sequence); + + if (status != F_true) { + printf("[ -------> ] --- [1] Failure at line=%lu sequence='%0x' (%llu).\n", line + 1, sequence, code); + } + + assert_int_equal(status, F_true); + } + + ++line; + + } while (bytes > 0); + + fclose(file_combining); + fclose(file_joiner); + } +} + +#ifdef __cplusplus +} // extern "C" +#endif diff --git a/level_0/f_utf/tests/unit/c/test-utf-character_is_combining_joiner.h b/level_0/f_utf/tests/unit/c/test-utf-character_is_combining_joiner.h new file mode 100644 index 000000000..d46168dc2 --- /dev/null +++ b/level_0/f_utf/tests/unit/c/test-utf-character_is_combining_joiner.h @@ -0,0 +1,20 @@ +/** + * FLL - Level 0 + * + * Project: UTF + * API Version: 0.8 + * Licenses: lgpl-2.1-or-later + * + * Test the function in the utf project. + */ +#ifndef _TEST__F_utf_character_is_combining_joiner_h +#define _TEST__F_utf_character_is_combining_joiner_h + +/** + * Test that the function works. + * + * @see f_utf_character_is_combining_joiner() + */ +extern void test__f_utf_character_is_combining_joiner__works(void **state); + +#endif // _TEST__F_utf_character_is_combining_joiner_h diff --git a/level_0/f_utf/tests/unit/c/test-utf-is_combining_joiner.c b/level_0/f_utf/tests/unit/c/test-utf-is_combining_joiner.c new file mode 100644 index 000000000..e9d10b2dd --- /dev/null +++ b/level_0/f_utf/tests/unit/c/test-utf-is_combining_joiner.c @@ -0,0 +1,100 @@ +#include "test-utf.h" +#include "test-utf-is_combining_joiner.h" + +#ifdef __cplusplus +extern "C" { +#endif + +void test__f_utf_is_combining_joiner__works(void **state) { + + { + FILE *file_combining = data__bytesequence_file_open__combining(); + FILE *file_joiner = data__bytesequence_file_open__joiner(); + + assert_non_null(file_combining); + assert_non_null(file_joiner); + + f_utf_char_t sequence = 0; + ssize_t bytes = 0; + long long code = 0; + + f_number_unsigned_t line = 0; + + do { + bytes = data__bytesequence_get_line(file_combining, &sequence, &code); + + if (bytes > 0) { + const uint8_t width = macro_f_utf_char_t_width(sequence); + char buffer[5] = { 0, 0, 0, 0, 0 }; + + buffer[0] = macro_f_utf_char_t_to_char_1(sequence); + + if (width > 1) { + buffer[1] = macro_f_utf_char_t_to_char_2(sequence); + + if (width > 2) { + buffer[2] = macro_f_utf_char_t_to_char_3(sequence); + + if (width > 3) { + buffer[3] = macro_f_utf_char_t_to_char_4(sequence); + } + } + } + + const f_status_t status = f_utf_is_combining_joiner(buffer, 5); + + if (status != F_true) { + printf("[ -------> ] --- [0] Failure at line=%lu sequence='%0x' (%llu).\n", line + 1, sequence, code); + } + + assert_int_equal(status, F_true); + } + + ++line; + + } while (bytes > 0); + + line = 0; + + do { + bytes = data__bytesequence_get_line(file_joiner, &sequence, &code); + + if (bytes > 0) { + const uint8_t width = macro_f_utf_char_t_width(sequence); + char buffer[5] = { 0, 0, 0, 0, 0 }; + + buffer[0] = macro_f_utf_char_t_to_char_1(sequence); + + if (width > 1) { + buffer[1] = macro_f_utf_char_t_to_char_2(sequence); + + if (width > 2) { + buffer[2] = macro_f_utf_char_t_to_char_3(sequence); + + if (width > 3) { + buffer[3] = macro_f_utf_char_t_to_char_4(sequence); + } + } + } + + const f_status_t status = f_utf_is_combining_joiner(buffer, 5); + + if (status != F_true) { + printf("[ -------> ] --- [1] Failure at line=%lu sequence='%0x' (%llu).\n", line + 1, sequence, code); + } + + assert_int_equal(status, F_true); + } + + ++line; + + } while (bytes > 0); + + fclose(file_combining); + fclose(file_joiner); + } +} + +#ifdef __cplusplus +} // extern "C" +#endif diff --git a/level_0/f_utf/tests/unit/c/test-utf-is_combining_joiner.h b/level_0/f_utf/tests/unit/c/test-utf-is_combining_joiner.h new file mode 100644 index 000000000..c507f67b4 --- /dev/null +++ b/level_0/f_utf/tests/unit/c/test-utf-is_combining_joiner.h @@ -0,0 +1,20 @@ +/** + * FLL - Level 0 + * + * Project: UTF + * API Version: 0.8 + * Licenses: lgpl-2.1-or-later + * + * Test the function in the utf project. + */ +#ifndef _TEST__F_utf_is_combining_joiner_h +#define _TEST__F_utf_is_combining_joiner_h + +/** + * Test that the function works. + * + * @see f_utf_is_combining_joiner() + */ +extern void test__f_utf_is_combining_joiner__works(void **state); + +#endif // _TEST__F_utf_is_combining_joiner_h diff --git a/level_0/f_utf/tests/unit/c/test-utf.c b/level_0/f_utf/tests/unit/c/test-utf.c index 73e0b0e97..bf2c58b12 100644 --- a/level_0/f_utf/tests/unit/c/test-utf.c +++ b/level_0/f_utf/tests/unit/c/test-utf.c @@ -28,6 +28,7 @@ int main(void) { cmocka_unit_test(test__f_utf_character_is_alphabetic_digit__works), cmocka_unit_test(test__f_utf_character_is_alphabetic_numeric__works), cmocka_unit_test(test__f_utf_character_is_combining__works), + cmocka_unit_test(test__f_utf_character_is_combining_joiner__works), cmocka_unit_test(test__f_utf_character_is_control__works), cmocka_unit_test(test__f_utf_character_is_control_picture__works), cmocka_unit_test(test__f_utf_character_is_digit__works), @@ -142,6 +143,7 @@ int main(void) { cmocka_unit_test(test__f_utf_is_alphabetic_digit__works), cmocka_unit_test(test__f_utf_is_alphabetic_numeric__works), cmocka_unit_test(test__f_utf_is_combining__works), + cmocka_unit_test(test__f_utf_is_combining_joiner__works), cmocka_unit_test(test__f_utf_is_control__works), cmocka_unit_test(test__f_utf_is_control_picture__works), cmocka_unit_test(test__f_utf_is_digit__works), diff --git a/level_0/f_utf/tests/unit/c/test-utf.h b/level_0/f_utf/tests/unit/c/test-utf.h index bff5053de..a8c5cc98d 100644 --- a/level_0/f_utf/tests/unit/c/test-utf.h +++ b/level_0/f_utf/tests/unit/c/test-utf.h @@ -37,6 +37,7 @@ #include "test-utf-character_is_alphabetic_digit.h" #include "test-utf-character_is_alphabetic_numeric.h" #include "test-utf-character_is_combining.h" +#include "test-utf-character_is_combining_joiner.h" #include "test-utf-character_is_control.h" #include "test-utf-character_is_control_picture.h" #include "test-utf-character_is_digit.h" @@ -104,6 +105,7 @@ #include "test-utf-is_alphabetic_digit.h" #include "test-utf-is_alphabetic_numeric.h" #include "test-utf-is_combining.h" +#include "test-utf-is_combining_joiner.h" #include "test-utf-is_control.h" #include "test-utf-is_control_picture.h" #include "test-utf-is_digit.h" diff --git a/level_1/fl_fss/c/fss/basic.c b/level_1/fl_fss/c/fss/basic.c index fdfa200c7..ff5a2b7a9 100644 --- a/level_1/fl_fss/c/fss/basic.c +++ b/level_1/fl_fss/c/fss/basic.c @@ -18,8 +18,7 @@ extern "C" { #endif // _di_level_1_parameter_checking_ f_fss_skip_past_space(buffer, range, state); - if (F_status_is_error(state->status)) return; - if (state->status == F_data_not) return; + if (F_status_is_error(state->status) || state->status == F_data_not) return; if (state->status == F_okay_eol) { ++range->start; diff --git a/level_1/fl_fss/c/fss/basic.h b/level_1/fl_fss/c/fss/basic.h index 20ccbdc05..eeeab0bbd 100644 --- a/level_1/fl_fss/c/fss/basic.h +++ b/level_1/fl_fss/c/fss/basic.h @@ -34,6 +34,8 @@ extern "C" { * * This will record where delimit placeholders exist but will not apply the delimits. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param buffer * The buffer to read from. * @param range @@ -99,6 +101,8 @@ extern "C" { * * The destination string may have NULLs. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param content * The string to write as (does not stop at NULLS, they are ignored and not written). * @param complete @@ -155,6 +159,8 @@ extern "C" { * * This will record where delimit placeholders exist but will not apply the delimits. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param buffer * The buffer to read from. * @param range @@ -234,6 +240,8 @@ extern "C" { * * The destination string may have NULLs. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param object * The string to write as (does not stop at NULLS, they are ignored and not written). * @param quote diff --git a/level_1/fl_fss/c/fss/basic_list.h b/level_1/fl_fss/c/fss/basic_list.h index 3097e1e39..5f74cd146 100644 --- a/level_1/fl_fss/c/fss/basic_list.h +++ b/level_1/fl_fss/c/fss/basic_list.h @@ -34,6 +34,8 @@ extern "C" { * * This will record where delimit placeholders exist but will not apply the delimits. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param buffer * The buffer to read from. * @param range @@ -109,6 +111,8 @@ extern "C" { * * The destination string may have NULLs. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param content * The string to write as (does not stop at NULLS, they are ignored and not written). * @param complete @@ -179,6 +183,8 @@ extern "C" { * * This will record where delimit placeholders exist but will not apply the delimits. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param buffer * The buffer to read from. * @param range @@ -253,6 +259,8 @@ extern "C" { * * The destination string may have NULLs. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param object * The string to write as (does not stop at NULLS, they are ignored and not written). * @param complete diff --git a/level_1/fl_fss/c/fss/embedded_list.h b/level_1/fl_fss/c/fss/embedded_list.h index 3e5a3ddd6..46b291c3b 100644 --- a/level_1/fl_fss/c/fss/embedded_list.h +++ b/level_1/fl_fss/c/fss/embedded_list.h @@ -74,6 +74,8 @@ extern "C" { * Therefore the object for this content will also need to be found.array[0].array[found.array[0].used].object. * The object at depth 0 will not be populated, but all nested objects will be populated at their respective depth along with the content. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param buffer * The buffer to read from. * @param range @@ -155,6 +157,8 @@ extern "C" { * * The destination string may have NULLs. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param content * The string to write as (does not stop at NULLS, they are ignored and not written). * @param complete @@ -230,6 +234,8 @@ extern "C" { * * This will record where delimit placeholders exist but will not apply the delimits. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param buffer * The buffer to read from. * @param range @@ -304,6 +310,8 @@ extern "C" { * * The destination string may have NULLs. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param object * The string to write as (does not stop at NULLS, they are ignored and not written). * @param complete diff --git a/level_1/fl_fss/c/fss/extended.c b/level_1/fl_fss/c/fss/extended.c index 2d64e5402..1ba037fe5 100644 --- a/level_1/fl_fss/c/fss/extended.c +++ b/level_1/fl_fss/c/fss/extended.c @@ -18,8 +18,7 @@ extern "C" { #endif // _di_level_1_parameter_checking_ f_fss_skip_past_space(buffer, range, state); - if (F_status_is_error(state->status)) return; - if (state->status == F_data_not) return; + if (F_status_is_error(state->status) || state->status == F_data_not) return; if (state->status == F_okay_eol) { ++range->start; diff --git a/level_1/fl_fss/c/fss/extended.h b/level_1/fl_fss/c/fss/extended.h index 1d3ae20fe..c57f5e168 100644 --- a/level_1/fl_fss/c/fss/extended.h +++ b/level_1/fl_fss/c/fss/extended.h @@ -34,6 +34,8 @@ extern "C" { * * This will record where delimit placeholders exist but will not apply the delimits. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param buffer * The buffer to read from. * @param range @@ -114,6 +116,8 @@ extern "C" { * * The destination string may have NULLs. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param content * The string to write as (does not stop at NULLS, they are ignored and not written). * @param quote @@ -183,6 +187,8 @@ extern "C" { * * This will record where delimit placeholders exist but will not apply the delimits. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param buffer * The buffer to read from. * @param range @@ -263,6 +269,8 @@ extern "C" { * * The destination string may have NULLs. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param object * The string to write as (does not stop at NULLS, they are ignored and not written). * @param quote diff --git a/level_1/fl_fss/c/fss/extended_list.c b/level_1/fl_fss/c/fss/extended_list.c index 4a88b779f..5da2e51d0 100644 --- a/level_1/fl_fss/c/fss/extended_list.c +++ b/level_1/fl_fss/c/fss/extended_list.c @@ -276,10 +276,17 @@ extern "C" { const f_number_unsigned_t destination_used = destination->used; - uint8_t is_comment = F_false; - uint8_t ends_on_eol = F_false; - uint8_t has_graph = F_false; - uint8_t do_prepend = prepend ? F_true : F_false; + #define ___fl_flag_is_comment___ 0x1 + #define ___fl_flag_has_graph___ 0x2 + #define ___fl_flag_is_comment_graph___ 0x3 + #define ___fl_flag_ends_on_eol___ 0x4 + #define ___fl_flag_do_prepend___ 0x8 + #define ___fl_flag_do_prepend_ends_on_eol___ 0xc + #define ___fl_flag_graph_combine___ 0x10 // Is a graph only due to combining character. + #define ___fl_flag_graph_joiner___ 0x20 // Is a graph only due to a joining character. + #define ___fl_flag_graph_joiner_combine___ 0x30 + + uint8_t flag = prepend ? ___fl_flag_do_prepend___ : 0x0; f_number_unsigned_t r = 0; f_number_unsigned_t start = 0; @@ -293,16 +300,16 @@ extern "C" { if (F_status_set_fine(state->status) == F_interrupt) break; } - if (content.string[range->start] == f_fss_slash_s.string[0] && !is_comment) { + if (content.string[range->start] == f_fss_slash_s.string[0] && !(flag & ___fl_flag_is_comment___)) { - if (do_prepend) { + if (flag & ___fl_flag_do_prepend___) { state->status = f_memory_array_increase_by(prepend->used, sizeof(f_char_t), (void **) &destination->string, &destination->used, &destination->size); if (F_status_is_error(state->status)) break; state->status = f_string_dynamic_append(*prepend, destination); if (F_status_is_error(state->status)) break; - do_prepend = F_false; + flag &= ~___fl_flag_do_prepend___; } else { state->status = f_memory_array_increase(state->step_large, sizeof(f_char_t), (void **) &destination->string, &destination->used, &destination->size); @@ -342,7 +349,7 @@ extern "C" { f_fss_skip_past_space(content, range, state); if (F_status_is_error(state->status)) break; - if (has_graph) { + if (flag & ___fl_flag_has_graph___) { // Do nothing. } else if (content.string[range->start] == f_fss_eol_s.string[0] || range->start >= content.used || range->start > range->stop) { @@ -358,7 +365,7 @@ extern "C" { if (F_status_is_error(state->status)) break; if (content.string[range->start] != f_fss_eol_s.string[0]) { - has_graph = F_true; + flag |= ___fl_flag_has_graph___; } continue; @@ -372,10 +379,10 @@ extern "C" { if (content.string[range->start] == f_fss_eol_s.string[0]) { destination->string[destination->used++] = f_fss_eol_s.string[0]; - ends_on_eol = F_true; + flag |= ___fl_flag_ends_on_eol___; } else { - ends_on_eol = F_false; + flag &= ~___fl_flag_ends_on_eol___; } range->start = start + 1; @@ -383,17 +390,17 @@ extern "C" { continue; } } - else if (!has_graph && content.string[range->start] == f_fss_extended_list_close_s.string[0] && !is_comment) { + else if (!(flag & ___fl_flag_is_comment_graph___) && content.string[range->start] == f_fss_extended_list_close_s.string[0]) { start = range->start++; - if (do_prepend) { + if (flag & ___fl_flag_do_prepend___) { state->status = f_string_dynamic_append(*prepend, destination); if (F_status_is_error(state->status)) break; - do_prepend = F_false; + flag &= ~___fl_flag_do_prepend___; } - has_graph = F_true; + flag |= ___fl_flag_has_graph___; f_fss_skip_past_space(content, range, state); if (F_status_is_error(state->status)) break; @@ -401,11 +408,10 @@ extern "C" { if (content.string[range->start] == f_fss_eol_s.string[0] || range->start >= content.used || range->start > range->stop) { if (content.string[range->start] == f_fss_eol_s.string[0] && prepend) { - do_prepend = F_true; - ends_on_eol = F_true; + flag |= ___fl_flag_do_prepend_ends_on_eol___; } else { - ends_on_eol = F_false; + flag &= ~___fl_flag_ends_on_eol___; } if (ignore && ignore->used) { @@ -448,48 +454,64 @@ extern "C" { continue; } - else if (content.string[range->start] == f_fss_comment_s.string[0] && !has_graph) { - is_comment = F_true; + else if (content.string[range->start] == f_fss_comment_s.string[0] && !(flag & ___fl_flag_has_graph___)) { + flag |= ___fl_flag_is_comment___; } else if (content.string[range->start] == f_fss_eol_s.string[0]) { - has_graph = F_false; - is_comment = F_false; + flag &= ~(flag & ___fl_flag_is_comment_graph___); } - else if (f_fss_is_graph(content, *range, state) == F_false) { - if (F_status_is_error(state->status)) break; + else if (f_fss_is_graph(content, *range, state) == F_true) { + flag |= ___fl_flag_has_graph___; } else { - has_graph = F_true; + if (F_status_is_error(state->status)) break; + + // TODO: flag |= ___fl_flag_graph_combine___; + // TODO: flag |= ___fl_flag_graph_joiner___; } if (content.string[range->start] != f_fss_placeholder_s.string[0]) { - if (do_prepend) { + if (flag & ___fl_flag_do_prepend___) { state->status = f_string_dynamic_append(*prepend, destination); if (F_status_is_error(state->status)) break; - do_prepend = F_false; + flag &= ~___fl_flag_do_prepend___; } if (content.string[range->start] == f_fss_eol_s.string[0] && prepend) { - do_prepend = F_true; - ends_on_eol = F_true; + flag |= ___fl_flag_do_prepend_ends_on_eol___; } else { - ends_on_eol = F_false; + flag &= ~___fl_flag_ends_on_eol___; } width = macro_f_utf_byte_width(content.string[range->start]); + // ___fl_flag_graph_combine___ and ___fl_flag_graph_joiner___ are only set if the characters can be found, so it is therefore safe to trust that range->start + width can be used. + if (flag & ___fl_flag_graph_combine___) { + width += macro_f_utf_byte_width(content.string[range->start + width]); + } + state->status = f_memory_array_increase_by(width, sizeof(f_char_t), (void **) &destination->string, &destination->used, &destination->size); if (F_status_is_error(state->status)) break; memcpy(destination->string + destination->used, content.string + range->start, width); destination->used += width; + + if (flag & ___fl_flag_graph_combine___) { + state->status = f_utf_buffer_increment(content, range, 2); + if (F_status_is_error(state->status)) break; + + flag &= ~___fl_flag_graph_combine___; + + continue; + } } state->status = f_utf_buffer_increment(content, range, 1); if (F_status_is_error(state->status)) break; + } // while if (F_status_is_error(state->status)) { @@ -507,7 +529,7 @@ extern "C" { return; } - if (!ends_on_eol) { + if (!(flag & ___fl_flag_ends_on_eol___)) { destination->string[destination->used++] = f_fss_eol_s.string[0]; } @@ -524,6 +546,16 @@ extern "C" { else { state->status = F_okay; } + + #undef ___fl_flag_is_comment___ + #undef ___fl_flag_has_graph___ + #undef ___fl_flag_is_comment_graph___ + #undef ___fl_flag_ends_on_eol___ + #undef ___fl_flag_do_prepend___ + #undef ___fl_flag_do_prepend_ends_on_eol___ + #undef ___fl_flag_graph_combine___ + #undef ___fl_flag_graph_joiner___ + #undef ___fl_flag_graph_joiner_combine___ } #endif // _di_fl_fss_extended_list_content_write_ diff --git a/level_1/fl_fss/c/fss/extended_list.h b/level_1/fl_fss/c/fss/extended_list.h index 200998901..54cca342a 100644 --- a/level_1/fl_fss/c/fss/extended_list.h +++ b/level_1/fl_fss/c/fss/extended_list.h @@ -36,6 +36,8 @@ extern "C" { * * This does not utilize recursion and has a max content depth of 1, see fss-0008 (Embedded List) for that purpose. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param buffer * The buffer to read from. * @param range @@ -115,6 +117,8 @@ extern "C" { * * The destination string may have NULLs. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param content * The string to write as (does not stop at NULLS, they are ignored and not written). * @param complete @@ -190,6 +194,8 @@ extern "C" { * * This will record where delimit placeholders exist but will not apply the delimits. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param buffer * The buffer to read from. * @param range @@ -264,6 +270,8 @@ extern "C" { * * The destination string may have NULLs. * + * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. + * * @param object * The string to write as (does not stop at NULLS, they are ignored and not written). * @param complete diff --git a/level_1/fl_fss/c/private-fss.c b/level_1/fl_fss/c/private-fss.c index f3c72e997..20741781f 100644 --- a/level_1/fl_fss/c/private-fss.c +++ b/level_1/fl_fss/c/private-fss.c @@ -9,14 +9,7 @@ extern "C" { void private_fl_fss_basic_or_extended_read(const f_string_static_t buffer, const uint8_t flag, f_range_t * const range, f_range_t * const found, uint8_t * const quote, f_number_unsigneds_t * const delimits, f_state_t * const state) { f_fss_skip_past_space(buffer, range, state); - - if (F_status_is_error(state->status) || state->status == F_data_not) { - if (F_status_set_fine(state->status) == F_complete_not_utf_start) { - f_fss_skip_past_combining(buffer, range, state); - if (F_status_is_error(state->status) || state->status == F_data_not) return; - } - else return; - } + if (F_status_is_error(state->status) || state->status == F_data_not) return; if (state->status == F_okay_eol) { // Move the start position to after the EOL. @@ -58,7 +51,7 @@ extern "C" { } // Handle quote support. - f_char_t quote_found = 0; + f_char_t quote_found = 0; // TODO: check if quote has combining characters? if (quote) *quote = f_fss_quote_type_none_e; @@ -155,6 +148,7 @@ extern "C" { f_number_unsigned_t first_slash = 0; f_number_unsigned_t slash_count = 0; f_number_unsigned_t location = 0; + f_number_unsigned_t previous = 0; while (range->start <= range->stop && range->start < buffer.used) { @@ -297,10 +291,33 @@ extern "C" { } } // while - if (f_fss_is_graph(buffer, *range, state) == F_false) { + if (f_fss_is_graph(buffer, *range, state) != F_true) { if (F_status_is_error(state->status)) return; + + // Anything other than a new line must be checked if a combining or joiner is in the next character. + if (buffer.string[range->start] != f_fss_eol_s.string[0] && range->start <= range->stop && range->start < buffer.used) { + previous = range->start; + + state->status = f_utf_buffer_increment(buffer, range, 1); + if (F_status_is_error(state->status)) return; + + state->status = f_utf_is_combining_joiner( + buffer.string + range->start, + (range->stop - range->start) + 1 > buffer.used - range->start + ? buffer.used - range->start + : (range->stop - range->start) + 1 + ); + + // When status is F_true, then this is a combining or joiner that causes the previous character to be treated as a graph. + if (state->status == F_false) { + range->start = previous; + } else if (F_status_is_error(state->status)) { + return; + } + } } - else { + + if (state->status == F_true) { while (range->start <= range->stop && range->start < buffer.used && buffer.string[range->start] != f_fss_eol_s.string[0]) { if (state->interrupt) { @@ -637,14 +654,7 @@ extern "C" { void private_fl_fss_basic_write(const uint8_t flag, const f_string_static_t object, const uint8_t quote, f_range_t * const range, f_string_dynamic_t * const destination, f_state_t * const state, void * const internal) { f_fss_skip_past_space(object, range, state); - - if (F_status_is_error(state->status) || state->status == F_data_not) { - if (F_status_set_fine(state->status) == F_complete_not_utf_start) { - f_fss_skip_past_combining(object, range, state); - if (F_status_is_error(state->status) || state->status == F_data_not) return; - } - else return; - } + if (F_status_is_error(state->status) || state->status == F_data_not) return; if (state->status == F_okay_eos) { state->status = F_data_not_eos; diff --git a/level_1/fl_fss/c/private-fss.h b/level_1/fl_fss/c/private-fss.h index 3efe1c0e9..6a5ce2009 100644 --- a/level_1/fl_fss/c/private-fss.h +++ b/level_1/fl_fss/c/private-fss.h @@ -70,6 +70,7 @@ extern "C" { * * Errors (with error bit) from: f_memory_array_increase(). * Errors (with error bit) from: f_memory_array_increase_by(). + * Errors (with error bit) from: f_fss_is_combining(). * Errors (with error bit) from: f_fss_is_graph(). * Errors (with error bit) from: f_fss_is_space(). * Errors (with error bit) from: f_fss_is_zero_width(). @@ -77,9 +78,11 @@ extern "C" { * Errors (with error bit) from: f_fss_skip_past_delimit(). * Errors (with error bit) from: f_fss_skip_past_space(). * Errors (with error bit) from: f_utf_buffer_increment(). + * Errors (with error bit) from: f_utf_is_combining_joiner(). * * @see f_memory_array_increase() * @see f_memory_array_increase_by() + * @see f_fss_is_combining() * @see f_fss_is_graph() * @see f_fss_is_space() * @see f_fss_is_zero_width() @@ -87,6 +90,7 @@ extern "C" { * @see f_fss_skip_past_delimit() * @see f_fss_skip_past_space() * @see f_utf_buffer_increment() + * @see f_utf_is_combining_joiner() * @see fl_fss_basic_object_read() * @see fl_fss_extended_object_read() * @see fl_fss_extended_content_read() diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_content_read.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_content_read.3 index 4347ad6e9..47e3422b6 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_content_read.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_content_read.3 @@ -19,6 +19,8 @@ Read an fss-0000 content. .PP This will record where delimit placeholders exist but will not apply the delimits. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_content_write.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_content_write.3 index b6273cc20..a55752616 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_content_write.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_content_write.3 @@ -21,6 +21,8 @@ This will write the given string range as a valid content. Anything within this .PP The destination string may have NULLs. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_content_read.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_content_read.3 index 486cfafc9..5ffdbc20f 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_content_read.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_content_read.3 @@ -20,6 +20,8 @@ Read an fss-0002 Content from a given string. .PP This will record where delimit placeholders exist but will not apply the delimits. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_content_write.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_content_write.3 index 05dd18388..9014fe7db 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_content_write.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_content_write.3 @@ -22,6 +22,8 @@ This will write the given string range as a valid Content. Anything within this .PP The destination string may have NULLs. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_object_read.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_object_read.3 index 612a51a52..3dda6ce4b 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_object_read.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_object_read.3 @@ -19,6 +19,8 @@ Read an fss-0002 Object. .PP This will record where delimit placeholders exist but will not apply the delimits. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_object_write.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_object_write.3 index a5fc38f16..ce215d25f 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_object_write.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_list_object_write.3 @@ -21,6 +21,8 @@ This will write the given string range as a valid Object. Anything within this r .PP The destination string may have NULLs. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_object_read.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_object_read.3 index 5ff3261c1..eaa686ac8 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_object_read.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_object_read.3 @@ -20,6 +20,8 @@ Read an fss-0000 object. .PP This will record where delimit placeholders exist but will not apply the delimits. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_object_write.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_object_write.3 index d86e267c6..96e78a19f 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_object_write.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_basic_object_write.3 @@ -22,6 +22,8 @@ This will write the given string range as a valid object. Anything within this r .PP The destination string may have NULLs. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_content_read.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_content_read.3 index 412892192..cfb8f6fe8 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_content_read.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_content_read.3 @@ -22,6 +22,8 @@ This will record where delimit placeholders exist but will not apply the delimit .PP This operates on the assumption that found.array[0].array[found.array[0].used].content is the current content being processed. Therefore the object for this content will also need to be found.array[0].array[found.array[0].used].object. The object at depth 0 will not be populated, but all nested objects will be populated at their respective depth along with the content. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP The state.data must be of type fl_fss_data_embedded_list_t with all properties as valid. .PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_content_write.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_content_write.3 index 69f6a081a..c8269e7f9 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_content_write.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_content_write.3 @@ -23,6 +23,8 @@ This will write the given string range as a valid content. Anything within this .PP The destination string may have NULLs. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_object_read.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_object_read.3 index 1f9259762..d25ca6fcb 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_object_read.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_object_read.3 @@ -19,6 +19,8 @@ Read an fss-0008 object. .PP This will record where delimit placeholders exist but will not apply the delimits. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_object_write.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_object_write.3 index e9b32ee73..d70dc4f71 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_object_write.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_embedded_list_object_write.3 @@ -21,6 +21,8 @@ This will write the given string range as a valid object. Anything within this r .PP The destination string may have NULLs. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_content_read.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_content_read.3 index 2a484530e..035bc31bc 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_content_read.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_content_read.3 @@ -20,6 +20,8 @@ Read an fss-0001 content. .PP This will record where delimit placeholders exist but will not apply the delimits. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP f_fss_state_quote_not_e: Explicitly disable quotes, treating quotes and their respective delimits as normal characters. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_content_write.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_content_write.3 index d69128906..8e5bb612b 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_content_write.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_content_write.3 @@ -22,6 +22,8 @@ This will write the given string range as a valid content. Anything within this .PP The destination string may have NULLs. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_content_read.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_content_read.3 index e80f908f4..fc7fed5fb 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_content_read.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_content_read.3 @@ -23,6 +23,8 @@ This will record where delimit placeholders exist but will not apply the delimit .PP This does not utilize recursion and has a max content depth of 1, see fss-0008 (Embedded List) for that purpose. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_content_write.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_content_write.3 index a8419ffb1..64ff92455 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_content_write.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_content_write.3 @@ -23,6 +23,8 @@ This will write the given string range as a valid content. Anything within this .PP The destination string may have NULLs. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_object_read.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_object_read.3 index f2380708d..3bd9e4906 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_object_read.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_object_read.3 @@ -19,6 +19,8 @@ Read an fss-0003 object. .PP This will record where delimit placeholders exist but will not apply the delimits. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_object_write.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_object_write.3 index 57ff824d7..90026efd4 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_object_write.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_list_object_write.3 @@ -21,6 +21,8 @@ This will write the given string range as a valid object. Anything within this r .PP The destination string may have NULLs. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_object_read.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_object_read.3 index 42178395f..7edaa1355 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_object_read.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_object_read.3 @@ -20,6 +20,8 @@ Read an fss-0001 object. .PP This will record where delimit placeholders exist but will not apply the delimits. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_object_write.3 b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_object_write.3 index a5fb24a50..1e0ed5347 100644 --- a/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_object_write.3 +++ b/level_1/fl_fss/data/documentation/man/man3/fl_fss_extended_object_write.3 @@ -22,6 +22,8 @@ This will write the given string range as a valid object. Anything within this r .PP The destination string may have NULLs. .PP +Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters. +.PP When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored. .PP Must not be NULL. diff --git a/specifications/fss.txt b/specifications/fss.txt index 88c176bd3..53a570338 100644 --- a/specifications/fss.txt +++ b/specifications/fss.txt @@ -1,7 +1,7 @@ # fss-0002 iki-0000 # # license open-standard-license-1.0-or-later -# version 2026/05/31 +# version 2026/06/20 # # This file (assumed to be named fss.txt) can be more easily read using the following iki_read commands: # iki_read fss.txt +Q -w -rrrrrrrr anti-KISS anti-KISS ASCII ASCII BOM BOM FSS FSS KISS KISS UTF-8 UTF-8 URL URL XML XML -WWW character "'" "'" code '"' '"' italic '"' '"' @@ -48,17 +48,21 @@ Featureless Settings Specifications: These specifications refer to characters that have no printable representation as italic:"non-printable". White spaces characters that are printable, such as tabs and spaces, must be considered the same type for the purposes of parsing. Non-printing white spaces characters (zero-width characters) are ignored, are treated as placeholders for processing with the exception of combining characters. - White spaces that use combining characters result in printable characters and the resulting combination is treated as not white space. + Using a combining character automatically causes the preceding character to be considered a graph character. + White space characters that use combining characters are treated as a graph character. + Zero-width characters that use combining characters are treated as a graph character. + The new line character:"\n" (unicode:"U+000A") is an exception to this combining characters rule and is always treated as a new line without regard to any combining character. A specification may override this handling of combining characters through explicit definitions. - Zero-width characters that use combining characters are treated as non-printing characters and are skipped. + For the purposes of processing, the use of a joiner character (unicode:"U+200D" and unicode:"U+2060") is to be treated in the same way as a combining character. + This is considered the case even if the joiner joins two non-graph characters. In terms of processing, it is recommended that the code:"NULL" character is not considered the end of a string, but this is only a suggestion. - Any specification may chose to limit, restrict, or otherwise prohibit special Unicode characters such as combining characters or zero-width characters. + Any specification may chose to limit, restrict, or otherwise prohibit special Unicode characters such as combining characters, zero-width, and joiner characters. - The current behavior of existing tools treat new line character:"\n" (unicode:"U+000A") the same regardless if a combining character follows this. + The current behavior of common tools treat new line character:"\n" (unicode:"U+000A") the same regardless if a combining character follows this. As an exception to the above rules (in order to avoid confusion with how existing software handles this behavior) a new line character:"\n" (unicode:"U+000A") is always treated as a such even if followed by a combining character. In this situation the combining characters at the start can be considered invalid, however the standard behavior is to ignore them and not treat them as part of the data. Should the data be sensitive in nature or require strict structure, then treating this situation as an error invalid is recommended. - Should tools, some time in the future, decide to combine onto character:"\n" (unicode:"U+000A"), then this standard can (and should) be reveiwed and updated to a more proper behavior of actually combining onto the control character. + Should tools, some time in the future, decide to combine onto character:"\n" (unicode:"U+000A"), then this standard can (and should) be reviewed and potentially updated to a more proper behavior of actually combining onto the control character. Unless otherwise specified, new lines designate the potential start (or end) of an Object or Content. @@ -75,6 +79,8 @@ Featureless Settings Specifications: "Object 1\" is an unterminated object due to the escaped closing quote. "Object 1\\" has content starting at the 'has', with an Object named "Object 1\". + Unless otherwise specified, quotes with combining or joining characters are not to be considered a quote for these purposes. + Unless otherwise specified, character/data delimits are performed only when required and not unilaterally. In the case of Objects, delimits would only apply when that Object could be potentially identified as an Object. For example, code:"FSS-0001 (Extended)" needs quotes to group parts that include spaces, if there is no initial quote, then a quote following the data italic:"must not" be delimited. -- 2.52.0