The main FSS specification with clarifications.
I'm generally in favor of performing calculations once.
However, I decided to try reducing the memory allocation and deallocation cost of variables by applying the simple math operations each time for some functions where this is performed only once.
This has a cost of some code readability, however.
The affected functions are single purpose and simple making the cost rather moot.
Begin changing the logic to add a new feature to solve a problem with insufficient compliance to the Unicode standards.
The joiners need to be handled.
The combining combines into what is effectively a graph character.
Treat the joiner characters the same, even though one can join two non-printing characters.
New lines must ignore joiners (I have seen this as the standard behavior by a lot of software and decided to go with that to avoid problems).
The is_combining, is_joiner, and is_combining_joiner are new features added to help solve these problems.
A lot of the logic is incomplete as of this commit.
I didn't want to add a cost of auto-checking every character after the first.
This means that I cannot have the is_space, is_graph, or other such functions automnate this.
Therefore, the logic must be changed to check for these spaces.
The code will become more complex but this will not incur as much of a performance cost compared to checking each character 2x+ times.
I also decided not to continue on after the first combining and joiner characters.
The combining already makes something a graph.
The joiner requires more searched and is highly subjective.
Therefore, make life easier and just treat all joiners as graph-making for the purposes of processing and syntax.
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
//#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
//#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_is_emoji_
#define _di_f_utf_is_fragment_
#define _di_f_utf_is_graph_
-//#define _di_f_utf_is_joiner_
+#define _di_f_utf_is_joiner_
#define _di_f_utf_is_numeric_
#define _di_f_utf_is_phonetic_
#define _di_f_utf_is_private_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_is_emoji_
#define _di_f_utf_is_fragment_
#define _di_f_utf_is_graph_
-//#define _di_f_utf_is_joiner_
+#define _di_f_utf_is_joiner_
#define _di_f_utf_is_numeric_
#define _di_f_utf_is_phonetic_
#define _di_f_utf_is_private_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_is_emoji_
#define _di_f_utf_is_fragment_
#define _di_f_utf_is_graph_
-//#define _di_f_utf_is_joiner_
+#define _di_f_utf_is_joiner_
#define _di_f_utf_is_numeric_
#define _di_f_utf_is_phonetic_
#define _di_f_utf_is_private_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_is_emoji_
#define _di_f_utf_is_fragment_
#define _di_f_utf_is_graph_
-//#define _di_f_utf_is_joiner_
+#define _di_f_utf_is_joiner_
#define _di_f_utf_is_numeric_
#define _di_f_utf_is_phonetic_
#define _di_f_utf_is_private_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
#define _di_f_utf_character_is_alphabetic_numeric_
#define _di_f_utf_character_is_ascii_
//#define _di_f_utf_character_is_combining_
+#define _di_f_utf_character_is_combining_joinger_
//#define _di_f_utf_character_is_control_
#define _di_f_utf_character_is_control_code_
#define _di_f_utf_character_is_control_format_
#define _di_f_utf_is_alphabetic_numeric_
#define _di_f_utf_is_ascii_
//#define _di_f_utf_is_combining_
+#define _di_f_utf_is_combining_joiner_
//#define _di_f_utf_is_control_
#define _di_f_utf_is_control_code_
#define _di_f_utf_is_control_format_
return F_false;
}
- f_number_unsigned_t width_max = (range.stop - range.start) + 1;
+ state->status = f_utf_is_combining(
+ buffer.string + range.start,
+ (range.stop - range.start) + 1 > buffer.used - range.start
+ ? buffer.used - range.start
+ : (range.stop - range.start) + 1
+ );
- if (width_max > buffer.used - range.start) {
- width_max = buffer.used - range.start;
- }
-
- state->status = f_utf_is_combining(buffer.string + range.start, width_max);
if (state->status == F_true) return F_true;
f_fss_fail_utf_to_false(state);
return F_false;
}
- f_number_unsigned_t width_max = (range.stop - range.start) + 1;
-
- if (width_max > buffer.used - range.start) {
- width_max = buffer.used - range.start;
- }
+ state->status = f_utf_is_graph(
+ buffer.string + range.start,
+ (range.stop - range.start) + 1 > buffer.used - range.start
+ ? buffer.used - range.start
+ : (range.stop - range.start) + 1
+ );
- state->status = f_utf_is_graph(buffer.string + range.start, width_max);
if (state->status == F_true) return F_true;
f_fss_fail_utf_to_false(state);
return F_false;
}
- f_number_unsigned_t width_max = (range.stop - range.start) + 1;
+ state->status = f_utf_is_zero_width(
+ buffer.string + range.start,
+ (range.stop - range.start) + 1 > buffer.used - range.start
+ ? buffer.used - range.start
+ : (range.stop - range.start) + 1
+ );
- if (width_max > buffer.used - range.start) {
- width_max = buffer.used - range.start;
+ if (state->status == F_true) {
+ state->status = F_space;
+
+ return F_false;
}
- state->status = f_utf_is_zero_width(buffer.string + range.start, width_max);
- if (state->status != F_false) return F_false;
+ if (state->status == F_false) {
+ state->status = f_utf_is_whitespace(
+ buffer.string + range.start,
+ (range.stop - range.start) + 1 > buffer.used - range.start
+ ? buffer.used - range.start
+ : (range.stop - range.start) + 1,
+ F_false
+ );
+
+ if (state->status == F_false) return F_false;
+ }
- f_fss_fail_utf_to_false(state);
+ if (state->status == F_true) {
+ state->status = f_utf_is_control(
+ buffer.string + range.start,
+ (range.stop - range.start) + 1 > buffer.used - range.start
+ ? buffer.used - range.start
+ : (range.stop - range.start) + 1
+ );
+
+ if (state->status == F_true) {
+ state->status = F_control;
+
+ // Some ASCII control characters are space characters.
+ if (!macro_f_utf_byte_width_is(buffer.string[range.start])) {
+ return F_true;
+ }
- state->status = f_utf_is_whitespace(buffer.string + range.start, width_max, F_false);
- if (state->status == F_true) return F_true;
+ return F_false;
+ }
+ else if (state->status == F_false) {
+ return F_true;
+ }
+ }
f_fss_fail_utf_to_false(state);
- if (state->status == F_false) {
- state->status = f_utf_is_control(buffer.string + range.start, width_max);
- if (state->status == F_true) return F_true;
-
- f_fss_fail_utf_to_false(state);
- }
-
return F_false;
}
#endif // _di_f_fss_is_space_
return F_false;
}
- f_number_unsigned_t width_max = (range.stop - range.start) + 1;
+ state->status = f_utf_is_zero_width(
+ buffer.string + range.start,
+ (range.stop - range.start) + 1 > buffer.used - range.start
+ ? buffer.used - range.start
+ : (range.stop - range.start) + 1
+ );
- if (width_max > buffer.used - range.start) {
- width_max = buffer.used - range.start;
- }
-
- state->status = f_utf_is_zero_width(buffer.string + range.start, width_max);
if (state->status == F_true) return F_true;
f_fss_fail_utf_to_false(state);
if (buffer.string[range->start] != f_fss_placeholder_s.string[0]) break;
} // for
- uint8_t width = 0;
- f_number_unsigned_t width_max = (range->stop - range->start) + 1;
+ if (macro_f_utf_byte_width_is(buffer.string[range->start])) {
+ if (range->start + macro_f_utf_byte_width(buffer.string[range->start]) > buffer.used) {
+ state->status = F_status_set_error(F_complete_not_utf_eos);
- if (width_max > buffer.used - range->start) {
- width_max = buffer.used - range->start;
- }
+ f_fss_fail_utf(state);
- // Check that the first character is not a combining character.
- state->status = f_utf_is_combining(buffer.string + range->start, width_max);
+ return;
+ }
- f_fss_fail_utf_to_false(state);
- if (F_status_is_error(state->status)) return;
+ if (range->start + (macro_f_utf_byte_width(buffer.string[range->start]) - 1) > range->stop) {
+ state->status = F_status_set_error(F_complete_not_utf_stop);
- if (state->status == F_true) {
- state->status = F_is_not;
+ f_fss_fail_utf(state);
- return;
+ return;
+ }
}
for (;;) {
- width = macro_f_utf_byte_width(buffer.string[range->start]);
-
- if (width > 1) {
- if (range->start + width >= buffer.used) {
- state->status = F_status_set_error(F_complete_not_utf_eos);
-
- f_fss_fail_utf(state);
-
- return;
- }
-
- if (range->start + width > range->stop) {
- state->status = F_status_set_error(F_complete_not_utf_stop);
-
- f_fss_fail_utf(state);
-
- return;
- }
- }
-
- range->start += width;
-
if (range->start >= buffer.used) {
state->status = F_okay_eos;
return;
}
- width_max = (range->stop - range->start) + 1;
-
- if (width_max > buffer.used - range->start) {
- width_max = buffer.used - range->start;
- }
-
if (buffer.string[range->start] == f_fss_eol_s.string[0]) {
state->status = F_okay_eol;
continue;
}
- state->status = f_utf_is_combining(buffer.string + range->start, width_max);
+ state->status = f_utf_is_combining_joiner(
+ buffer.string + range->start,
+ (range->stop - range->start) + 1 > buffer.used - range->start
+ ? buffer.used - range->start
+ : (range->stop - range->start) + 1
+ );
f_fss_fail_utf_to_false(state);
if (state->status == F_false) break;
if (F_status_is_error(state->status)) return;
+
+ if (macro_f_utf_byte_width_is(buffer.string[range->start])) {
+ if (range->start + macro_f_utf_byte_width(buffer.string[range->start]) > buffer.used) {
+ state->status = F_status_set_error(F_complete_not_utf_eos);
+
+ f_fss_fail_utf(state);
+
+ return;
+ }
+
+ if (range->start + (macro_f_utf_byte_width(buffer.string[range->start]) - 1) > range->stop) {
+ state->status = F_status_set_error(F_complete_not_utf_stop);
+
+ f_fss_fail_utf(state);
+
+ return;
+ }
+ }
+
+ range->start += macro_f_utf_byte_width(buffer.string[range->start]);
} // for
state->status = F_okay;
if (buffer.string[range->start] != f_fss_placeholder_s.string[0]) break;
} // for
- uint8_t width = 0;
f_number_unsigned_t previous = range->start;
- f_number_unsigned_t width_max = (range->stop - range->start) + 1;
- if (width_max > buffer.used - range->start) {
- width_max = buffer.used - range->start;
+ if (macro_f_utf_byte_width_is(buffer.string[range->start])) {
+ if (range->start + macro_f_utf_byte_width(buffer.string[range->start]) > buffer.used) {
+ state->status = F_status_set_error(F_complete_not_utf_eos);
+
+ f_fss_fail_utf(state);
+
+ return;
+ }
+
+ if (range->start + (macro_f_utf_byte_width(buffer.string[range->start]) - 1) > range->stop) {
+ state->status = F_status_set_error(F_complete_not_utf_stop);
+
+ f_fss_fail_utf(state);
+
+ return;
+ }
}
- // Check that the first character is not a combining character.
- state->status = f_utf_is_combining(buffer.string + range->start, width_max);
+ // Check that the first character is not a combining character or a joiner.
+ state->status = f_utf_is_combining_joiner(
+ buffer.string + previous,
+ (range->stop - previous) + 1 > buffer.used - previous
+ ? buffer.used - previous
+ : (range->stop - previous) + 1
+ );
f_fss_fail_utf_to_false(state);
+
if (F_status_is_error(state->status)) return;
if (state->status == F_true) {
for (;;) {
- width_max = (range->stop - range->start) + 1;
+ if (range->start >= buffer.used) {
+ state->status = F_okay_eos;
- if (width_max > buffer.used - range->start) {
- width_max = buffer.used - range->start;
+ return;
+ }
+
+ if (range->start > range->stop) {
+ state->status = F_okay_stop;
+
+ return;
}
if (buffer.string[range->start] == f_fss_eol_s.string[0]) {
continue;
}
- state->status = f_utf_is_whitespace(buffer.string + range->start, width_max, F_false);
+ state->status = f_utf_is_whitespace(
+ buffer.string + range->start,
+ (range->stop - range->start) + 1 > buffer.used - range->start
+ ? buffer.used - range->start
+ : (range->stop - range->start) + 1,
+ F_false
+ );
f_fss_fail_utf_to_false(state);
if (state->status == F_false) {
- state->status = f_utf_is_control(buffer.string + range->start, width_max);
+ state->status = f_utf_is_combining_joiner(
+ buffer.string + range->start,
+ (range->stop - range->start) + 1 > buffer.used - range->start
+ ? buffer.used - range->start
+ : (range->stop - range->start) + 1
+ );
f_fss_fail_utf_to_false(state);
+ // This is combining, so any previous white space is no longer considered white space as such.
+ if (state->status == F_true) {
+ range->start = previous;
+
+ break;
+ }
+
if (state->status == F_false) {
- state->status = f_utf_is_combining(buffer.string + range->start, width_max);
+ state->status = f_utf_is_control(
+ buffer.string + range->start,
+ (range->stop - range->start) + 1 > buffer.used - range->start
+ ? buffer.used - range->start
+ : (range->stop - range->start) + 1
+ );
f_fss_fail_utf_to_false(state);
if (state->status == F_false) {
- state->status = f_utf_is_zero_width(buffer.string + range->start, width_max);
+ state->status = f_utf_is_zero_width(
+ buffer.string + range->start,
+ (range->stop - range->start) + 1 > buffer.used - range->start
+ ? buffer.used - range->start
+ : (range->stop - range->start) + 1
+ );
f_fss_fail_utf_to_false(state);
if (state->status == F_false) break;
}
- else {
- // This is combining, so any previous white space is no longer considered white space as such.
- range->start = previous;
-
- // Ensure previous is not a place holder.
- while (range->start && buffer.string[range->start] == f_fss_placeholder_s.string[0]) --range->start;
-
- break;
- }
}
}
previous = range->start;
- width = macro_f_utf_byte_width(buffer.string[range->start]);
-
- if (width > 1) {
- if (range->start + width >= buffer.used) {
+ if (macro_f_utf_byte_width_is(buffer.string[previous])) {
+ if (previous + macro_f_utf_byte_width(buffer.string[previous]) >= buffer.used) {
state->status = F_status_set_error(F_complete_not_utf_eos);
f_fss_fail_utf(state);
return;
}
- if (range->start + width > range->stop) {
+ if (previous + macro_f_utf_byte_width(buffer.string[previous]) > range->stop) {
state->status = F_status_set_error(F_complete_not_utf_stop);
f_fss_fail_utf(state);
}
}
- range->start += width;
-
- if (range->start >= buffer.used) {
- state->status = F_okay_eos;
-
- return;
- }
-
- if (range->start > range->stop) {
- state->status = F_okay_stop;
-
- return;
- }
+ range->start += macro_f_utf_byte_width(buffer.string[previous]);
} // for
state->status = F_okay;
* A state for providing flags and handling interrupts during long running operations.
*
* This alters state.status:
- * F_true if the character in the buffer is a combining character.
+ * F_data_not on success but nothing is processed.
* F_false if the character in the buffer is not a combining character.
+ * F_true if the character in the buffer is a combining character.
*
* F_parameter (with error bit) if a parameter is invalid.
*
* Errors (with error bit) from: f_utf_is_combining().
*
* @return
- * F_true if the character in the buffer is a combining character.
* F_false if the character in the buffer is not a combining character.
* F_false is returned on any error.
+ * F_true if the character in the buffer is a combining character.
*
* @see f_utf_is_combining()
*/
/**
* Identify whether or not a character in the buffer is a graph (ASCII or UTF-8) character.
*
+ * This does not check characters beyond the current one to check for combining or joining characters.
+ *
* @param buffer
* The string to process.
* @param range
* A state for providing flags and handling interrupts during long running operations.
*
* This alters state.status:
- * F_true if the character in the buffer is a graph character.
+ * F_data_not on success but nothing is processed.
* F_false if the character in the buffer is not a graph character.
+ * F_true on success and is a graph character.
*
* F_parameter (with error bit) if a parameter is invalid.
*
* Errors (with error bit) from: f_utf_is_graph().
*
* @return
- * F_true if the character in the buffer is a graph character.
* F_false if the character in the buffer is not a graph character.
* F_false is returned on any error.
+ * F_true if the character in the buffer is a graph character.
*
* @see f_utf_is_graph()
*/
#endif // _di_f_fss_is_graph_
/**
- * Identify whether or not a character in the buffer is a non-zero-width whitespace or non-zero-width control (ASCII or UTF-8) character.
+ * Identify whether or not a character in the buffer is a non-zero-width white space or non-zero-width control (ASCII or UTF-8) character.
+ *
+ * This does not check characters beyond the current one to check for combining or joining characters.
*
* @param buffer
* The string to process.
* A state for providing flags and handling interrupts during long running operations.
*
* This alters state.status:
- * F_okay on success.
* F_data_not on success but nothing is processed.
+ * F_false if the character in the buffer is not a space character.
+ * F_space if the character is a zero width space (F_false is returned by this function).
+ * F_control if the character is a control character (F_false is returned by this function when zero-width or F_true when is both a space and a control character).
+ * F_true on success and is a space character.
+ *
* F_parameter (with error bit) if a parameter is invalid.
*
* Errors (with error bit) from: f_utf_is_control().
* Errors (with error bit) from: f_utf_is_zero_width().
*
* @return
- * F_true if the character in the buffer is a space character.
* F_false if the character in the buffer is not a space character.
* F_false is returned on any error.
+ * F_true if the character in the buffer is a space character.
*
* @see f_utf_is_control()
* @see f_utf_is_whitespace()
/**
* Identify whether or not a character in the buffer is a zero-width (ASCII or UTF-8) character.
*
+ * This does not check characters beyond the current one to check for combining or joining characters.
+ *
* The NULL character (U+0000) is a zero-width character.
*
* @param buffer
* A state for providing flags and handling interrupts during long running operations.
*
* This alters state.status:
- * F_true if the character in the buffer is a zero-width character.
+ * F_data_not on success but nothing is processed.
* F_false if the character in the buffer is not a zero-width character.
+ * F_true if the character in the buffer is a zero-width character.
*
* F_parameter (with error bit) if a parameter is invalid.
*
* Errors (with error bit) from: f_utf_is_zero_width().
*
* @return
- * F_true if the character in the buffer is a zero-width character.
* F_false if the character in the buffer is not a zero-width character.
* F_false is returned on any error.
+ * F_true if the character in the buffer is a zero-width character.
*
* @see f_utf_is_zero_width()
*/
#endif // _di_f_fss_seek_to_eol_
/**
- * Skip past all combining characters.
+ * Skip past all combining and joiner characters.
*
- * This will do nothing if the first character is not a combining character.
+ * This will do nothing if the first character is not a combining character and is not a joiner character.
*
* This skips past place holders.
*
* This alters state.status:
* F_okay on success.
* F_data_not on success but buffer.used is 0, initial range.start is greater than range.stop, or initial range.start is greater than or equal to buffer.used.
- * F_is_not on success, but the first character is not a combining character.
* F_okay_eol on success and EOL was reached.
* F_okay_eos on success and EOS was reached.
* F_okay_stop on success and stop point was reached.
* F_complete_not_utf_stop (with error bit) if unable to get entire UTF-8 sequence due to stop point reached.
* F_parameter (with error bit) if a parameter is invalid.
*
- * Errors (with error bit) from: f_utf_is_combining().
+ * Errors (with error bit) from: f_utf_is_combining_joiner().
*
- * @see f_utf_is_combining()
+ * @see f_utf_is_combining_joiner()
*/
#ifndef _di_f_fss_skip_past_combining_
extern void f_fss_skip_past_combining(const f_string_static_t buffer, f_range_t * const range, f_state_t * const state);
*
* If the first character in the given range is a combining character, then this will not skip past anything.
* This is because combining characters apply from right to left.
+ * The caller should check the return status and handle accordingly in this situation.
*
* White space is checked for combining characters after the white space.
* If there is a combining character after, then the white space is not consider as such.
* F_okay_stop on success and stop point was reached.
*
* F_complete_not_utf_eos (with error bit) if unable to get entire UTF-8 sequence due to EOS.
- * F_complete_not_utf_start (with error bit) if the first character is a combining character.
+ * F_complete_not_utf_start (with error bit) if the first character is a combining or a joiner character.
* F_complete_not_utf_stop (with error bit) if unable to get entire UTF-8 sequence due to stop point reached.
* F_parameter (with error bit) if a parameter is invalid.
*
- * Errors (with error bit) from: f_utf_is_combining().
+ * Errors (with error bit) from: f_utf_is_combining_joiner().
* Errors (with error bit) from: f_utf_is_control().
* Errors (with error bit) from: f_utf_is_whitespace().
* Errors (with error bit) from: f_utf_is_zero_width().
*
- * @see f_utf_is_combining()
+ * @see f_utf_is_combining_joiner()
* @see f_utf_is_control()
* @see f_utf_is_whitespace()
* @see f_utf_is_zero_width()
build_libraries -lc -lcmocka
build_libraries-individual -lf_memory -lf_string -lf_type_array -lf_utf -lf_fss
-build_sources_program test-fss-apply_delimit.c test-fss-apply_delimit_range.c test-fss-count_lines.c test-fss-count_lines_range.c test-fss-fail_utf.c test-fss-fail_utf_to_false.c test-fss-is_combining.c test-fss-is_graph.c test-fss-is_space.c test-fss-is_zero_width.c test-fss-seek_to_eol.c test-fss-skip_past_delimit.c test-fss-skip_past_space.c
+build_sources_program test-fss-apply_delimit.c test-fss-apply_delimit_range.c test-fss-count_lines.c test-fss-count_lines_range.c test-fss-fail_utf.c test-fss-fail_utf_to_false.c test-fss-is_combining.c test-fss-is_graph.c test-fss-is_space.c test-fss-is_zero_width.c test-fss-seek_to_eol.c test-fss-skip_past_combining.c test-fss-skip_past_delimit.c test-fss-skip_past_space.c
build_sources_program test-fss-item_delete.c test-fss-item_destroy.c
build_sources_program test-fss-items_delete_callback.c test-fss-items_destroy_callback.c test-fss-itemss_delete_callback.c test-fss-itemss_destroy_callback.c
build_sources_program test-fss-named_delete.c test-fss-named_destroy.c
.PP
Identify whether or not a character in the buffer is a graph (ASCII or UTF-8) character.
.PP
-This alters state.status: F_true if the character in the buffer is a graph character. F_false if the character in the buffer is not a graph character.
+If the character is not a graph character but is followed by a combining character, then it is considered a graph character due to the combination result.
+.PP
+This alters state.status: F_true if the character in the buffer is a graph character or the character is followed by a combining character. F_false if the character in the buffer is not a graph character.
.PP
F_parameter (with error bit) if a parameter is invalid.
.PP
.RE
.SH RETURN VALUE
.PP
-F_true if the character in the buffer is a graph character. F_false if the character in the buffer is not a graph character. F_false is returned on any error.
+F_true if the character in the buffer is a graph character or the character is followed by a combining character. F_false if the character in the buffer is not a graph character. F_false is returned on any error.
.SH SEE ALSO
.PP
.nh
.PP
Identify whether or not a character in the buffer is a non-zero-width whitespace or non-zero-width control (ASCII or UTF-8) character.
.PP
+A valid white space (or control character) that is followed by a combining character is considered a non-space unless that control character is a new line. The new line is an exception due to the current handling of that character with combining characters as observed in common editors.
+.PP
This alters state.status: F_okay on success. F_data_not on success but nothing is processed. F_parameter (with error bit) if a parameter is invalid.
.PP
Errors (with error bit) from: f_utf_is_control(). Errors (with error bit) from: f_utf_is_whitespace(). Errors (with error bit) from: f_utf_is_zero_width().
.RE
.SH RETURN VALUE
.PP
-F_true if the character in the buffer is a space character. F_false if the character in the buffer is not a space character. F_false is returned on any error.
+F_true if the character in the buffer is a space character (and the following character, if any, is not a combining character). F_false if the character in the buffer is not a space character. F_false is returned on any error.
.SH SEE ALSO
.PP
.nh
F_false,
F_true,
F_false,
- F_false,
- F_false,
+ F_true,
+ F_true,
F_false,
};
const f_status_t status = f_fss_is_space(tests[i], range, &state_data);
if (status != expects[i]) {
- printf("[ -------> ] --- [00] Expected %s at index %d.\n", expects[i] ? "false" : "true", i);
+ printf("[ -------> ] --- [00] Expected %s at index %d for string '%s'.\n", expects[i] ? "true" : "false", i, tests[i].used ? tests[i].string : "");
}
assert_int_equal(status, expects[i]);
const f_status_t status = f_fss_is_space(f_string_empty_s, range, &state_data);
if (status) {
- printf("[ -------> ] --- [01] Expected false for empty string with range (1, 0).\n");
+ printf("[ -------> ] --- [01] Expected true for empty string with range (%lu, %lu).\n", range.start, range.stop);
}
assert_int_equal(status, F_false);
+
+ if (state_data.status != F_data_not) {
+ printf("[ -------> ] --- [02] Expected F_data_not in state.status for empty string with range (%lu, %lu).\n", range.start, range.stop);
+ }
+
+ assert_int_equal(state_data.status, F_data_not);
}
{
const f_status_t status = f_fss_is_space(f_string_empty_s, range, &state_data);
if (status) {
- printf("[ -------> ] --- [01] Expected false for empty string with (out of) range (%lu, %lu).\n", range.start, range.stop);
+ printf("[ -------> ] --- [03] Expected true for empty string with (out of) range (%lu, %lu).\n", range.start, range.stop);
}
assert_int_equal(status, F_false);
+
+ if (state_data.status != F_data_not) {
+ printf("[ -------> ] --- [04] Expected F_data_not in state.status for empty string with (out of) range (%lu, %lu).\n", range.start, range.stop);
+ }
+
+ assert_int_equal(state_data.status, F_data_not);
}
}
--- /dev/null
+#include "test-fss.h"
+#include "test-fss-skip_past_combining.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+void test__f_fss_skip_past_combining__parameter_checking(void **state) {
+
+ const f_string_static_t test = macro_f_string_static_t_initialize_1("test", 4, 0);
+ f_state_t state_data = f_state_t_initialize;
+
+ {
+ f_fss_skip_past_combining(test, 0, &state_data);
+
+ assert_int_equal(state_data.status, F_status_set_error(F_parameter));
+ }
+}
+
+void test__f_fss_skip_past_combining__returns_data_not(void **state) {
+
+ const f_string_static_t test = macro_f_string_static_t_initialize_1("test", 4, 0);
+ f_state_t state_data = f_state_t_initialize;
+
+ {
+ f_range_t range = f_range_t_initialize;
+
+ f_fss_skip_past_combining(f_string_empty_s, &range, &state_data);
+
+ assert_int_equal(state_data.status, F_data_not);
+ }
+
+ {
+ f_range_t range = f_range_t_initialize;
+
+ f_fss_skip_past_combining(test, &range, &state_data);
+
+ assert_int_equal(state_data.status, F_data_not);
+ }
+
+ {
+ f_range_t range = macro_f_range_t_initialize_1(0, test.used - 1);
+
+ f_fss_skip_past_combining(f_string_empty_s, &range, &state_data);
+
+ assert_int_equal(state_data.status, F_data_not);
+ }
+
+ {
+ f_range_t range = macro_f_range_t_initialize_1(test.used, test.used + 2);
+
+ f_fss_skip_past_combining(test, &range, &state_data);
+
+ assert_int_equal(state_data.status, F_data_not);
+ }
+}
+
+void test__f_fss_skip_past_combining__works(void **state) {
+
+ f_state_t state_data = f_state_t_initialize;
+
+ f_string_static_t tests[] = {
+
+ // Set 1
+ macro_f_string_static_t_initialize_1("́", 2, 0),
+ macro_f_string_static_t_initialize_1("\0́", 3, 0),
+ macro_f_string_static_t_initialize_1("\0\0\0́", 5, 0),
+ macro_f_string_static_t_initialize_1("\0\0\0́\0\0", 7, 0),
+
+ // Set 2
+ macro_f_string_static_t_initialize_1("́.", 3, 0),
+ macro_f_string_static_t_initialize_1("\0̃.", 4, 0),
+ macro_f_string_static_t_initialize_1("\0\0\0̃.", 6, 0),
+ macro_f_string_static_t_initialize_1("\0\0\0̃\0\0.", 8, 0),
+
+ // Set 3
+ macro_f_string_static_t_initialize_1("̠.", 3, 0),
+ macro_f_string_static_t_initialize_1("\0̠.", 4, 0),
+ macro_f_string_static_t_initialize_1("\0\0\0̠.", 6, 0),
+ macro_f_string_static_t_initialize_1("\0\0\0̠\0\0.", 8, 0),
+
+ // Set 4
+ macro_f_string_static_t_initialize_1("̠́.", 5, 0),
+ macro_f_string_static_t_initialize_1("\0̠́.", 6, 0),
+ macro_f_string_static_t_initialize_1("\0\0\0̠́.", 8, 0),
+ macro_f_string_static_t_initialize_1("\0\0\0̠́\0\0.", 10, 0),
+
+ // Set 5
+ macro_f_string_static_t_initialize_1("̠́́.", 7, 0),
+ macro_f_string_static_t_initialize_1("\0̠́́.", 8, 0),
+ macro_f_string_static_t_initialize_1("\0\0\0̠́́.", 10, 0),
+ macro_f_string_static_t_initialize_1("\0\0\0̠́́\0\0.", 12, 0),
+ };
+
+ uint16_t statuss[] = {
+
+ // Set 1
+ F_okay_eos,
+ F_okay_eos,
+ F_okay_eos,
+ F_okay_eos,
+
+ // Set 2
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+
+ // Set 3
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+
+ // Set 4
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+
+ // Set 5
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+ };
+
+ f_number_unsigned_t ranges[] = {
+
+ // Set 1
+ 2,
+ 3,
+ 5,
+ 7,
+
+ // Set 2
+ 2,
+ 3,
+ 5,
+ 7,
+
+ // Set 3
+ 2,
+ 3,
+ 5,
+ 7,
+
+ // Set 4
+ 4,
+ 5,
+ 7,
+ 9,
+
+ // Set 5
+ 6,
+ 7,
+ 9,
+ 11,
+ };
+
+ for (uint8_t i = 0; i < 20; ++i) {
+
+ f_range_t range = macro_f_range_t_initialize_1(0, tests[i].used - 1);
+
+ f_fss_skip_past_combining(tests[i], &range, &state_data);
+
+ if (state_data.status != statuss[i]) {
+ printf("[ -------> ] --- [00] Expected %u in state.status but got %u at index %d for string '%s'.\n", statuss[i], state_data.status, i, tests[i].used ? tests[i].string : "");
+ }
+
+ assert_int_equal(state_data.status, statuss[i]);
+
+ if (range.start != ranges[i]) {
+ printf("[ -------> ] --- [01] Expected %lu in range.start but got %lu at index %d for string '%s'.\n", ranges[i], range.start, i, tests[i].used ? tests[i].string : "");
+ }
+
+ assert_int_equal(range.start, ranges[i]);
+ } // for
+}
+
+#ifdef __cplusplus
+} // extern "C"
+#endif
--- /dev/null
+/**
+ * FLL - Level 0
+ *
+ * Project: FSS
+ * API Version: 0.8
+ * Licenses: lgpl-2.1-or-later
+ *
+ * Test skip past combining in the fss project.
+ */
+#ifndef _TEST__F_fss__skip_past_combining
+#define _TEST__F_fss__skip_past_combining
+
+/**
+ * Test that the function correctly fails on invalid parameter.
+ *
+ * @see f_fss_skip_past_combining()
+ */
+extern void test__f_fss_skip_past_combining__parameter_checking(void **state);
+
+/**
+ * Test that the function returns F_data_not.
+ *
+ * @see f_fss_skip_past_combining()
+ */
+extern void test__f_fss_skip_past_combining__returns_data_not(void **state);
+
+/**
+ * Test that the function works.
+ *
+ * @see f_fss_skip_past_combining()
+ */
+extern void test__f_fss_skip_past_combining__works(void **state);
+
+#endif // _TEST__F_fss__skip_past_combining
macro_f_string_static_t_initialize_1("\0\0\0test\0\0", 9, 0),
};
- f_number_unsigned_t expects[] = {
+ uint16_t statuss[] = {
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+ };
+
+ f_number_unsigned_t ranges[] = {
0,
1,
3,
f_fss_skip_past_delimit(tests[i], &range, &state_data);
- assert_int_equal(state_data.status, F_okay);
- assert_int_equal(range.start, expects[i]);
+ if (state_data.status != statuss[i]) {
+ printf("[ -------> ] --- [00] Expected %u in state.status but got %u at index %d for string '%s'.\n", statuss[i], state_data.status, i, tests[i].used ? tests[i].string : "");
+ }
+
+ assert_int_equal(state_data.status, statuss[i]);
+
+ if (range.start != ranges[i]) {
+ printf("[ -------> ] --- [01] Expected %lu in range.start but got %lu at index %d for string '%s'.\n", ranges[i], range.start, i, tests[i].used ? tests[i].string : "");
+ }
+
} // for
}
f_state_t state_data = f_state_t_initialize;
+ // Space followed by a zero-width joiner and a joined character with NULLs before joiner (U+0020 \0 \0 U+200D U+1F9D4).
+ char space_null_join[] = { 0x20, 0x0, 0x0, 0xe2, 0x80, 0x8d, 0xf0, 0x9f, 0xa7, 0x94 };
+
f_string_static_t tests[] = {
- macro_f_string_static_t_initialize_1("test", 4, 0),
- macro_f_string_static_t_initialize_1(" test", 5, 0),
- macro_f_string_static_t_initialize_1(" test", 7, 0),
- macro_f_string_static_t_initialize_1(" \0test ", 9, 0),
- macro_f_string_static_t_initialize_1(" \t\atest ", 9, 0),
- macro_f_string_static_t_initialize_1("\a\t $test ", 10, 0),
- macro_f_string_static_t_initialize_1(" ́test", 7, 0),
+ macro_f_string_static_t_initialize_1("test", 4, 0),
+ macro_f_string_static_t_initialize_1(" test", 5, 0),
+ macro_f_string_static_t_initialize_1(" test", 7, 0),
+ macro_f_string_static_t_initialize_1(" \0test ", 9, 0),
+ macro_f_string_static_t_initialize_1(" \t\atest ", 9, 0),
+ macro_f_string_static_t_initialize_1("\a\t $test ", 10, 0),
+ macro_f_string_static_t_initialize_1(" ́test", 7, 0), // Space followed by a combining character (U+0020 U+0301).
+ macro_f_string_static_t_initialize_1("test", 7, 0), // Starts with zero-width space (U+200B).
+ macro_f_string_static_t_initialize_1(" 🧔test", 12, 0), // Space followed by a zero-width joiner and a joined character (U+0020 U+200D U+1F9D4).
+ macro_f_string_static_t_initialize_1(" test", 8, 0), // Space followed by a word joiner and a joined character (U+0020 U+2060 U+0062).
+ macro_f_string_static_t_initialize_1("́test", 6, 0), // Starts with combining character (U+0301).
+ macro_f_string_static_t_initialize_1("́test", 6, 0), // Starts with joiner character (U+2060).
+ macro_f_string_static_t_initialize_1(" \0\0🧔test", 14, 0), // Space followed by a zero-width joiner and a joined character with NULLs after joiner (U+0020 U+200D \0 \0 U+1F9D4).
+ macro_f_string_static_t_initialize_1(space_null_join, 14, 0),
+ };
+
+ uint16_t statuss[] = {
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+ F_okay,
+ F_status_set_error(F_complete_not_utf_start),
+ F_status_set_error(F_complete_not_utf_start),
+ F_okay,
+ F_okay,
};
- f_number_unsigned_t expects[] = {
+ f_number_unsigned_t ranges[] = {
0,
1,
3,
3,
3,
3,
+ 0,
3,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
};
- for (uint8_t i = 0; i < 7; ++i) {
+ for (uint8_t i = 0; i < 14; ++i) {
f_range_t range = macro_f_range_t_initialize_1(0, tests[i].used - 1);
f_fss_skip_past_space(tests[i], &range, &state_data);
- assert_int_equal(state_data.status, F_okay);
- assert_int_equal(range.start, expects[i]);
+ if (state_data.status != statuss[i]) {
+ printf("[ -------> ] --- [00] Expected %u in state.status but got %u at index %d for string '%s'.\n", statuss[i], state_data.status, i, tests[i].used ? tests[i].string : "");
+ }
+
+ assert_int_equal(state_data.status, statuss[i]);
+
+ if (range.start != ranges[i]) {
+ printf("[ -------> ] --- [01] Expected %lu in range.start but got %lu at index %d for string '%s'.\n", ranges[i], range.start, i, tests[i].used ? tests[i].string : "");
+ }
+
+ assert_int_equal(range.start, ranges[i]);
} // for
}
cmocka_unit_test(test__f_fss_count_lines__returns_data_not),
cmocka_unit_test(test__f_fss_count_lines_range__returns_data_not),
cmocka_unit_test(test__f_fss_seek_to_eol__returns_data_not),
+ cmocka_unit_test(test__f_fss_skip_past_combining__returns_data_not),
cmocka_unit_test(test__f_fss_skip_past_delimit__returns_data_not),
cmocka_unit_test(test__f_fss_skip_past_space__returns_data_not),
cmocka_unit_test(test__f_fss_is_space__works),
cmocka_unit_test(test__f_fss_is_zero_width__works),
cmocka_unit_test(test__f_fss_seek_to_eol__works),
+ cmocka_unit_test(test__f_fss_skip_past_combining__works),
cmocka_unit_test(test__f_fss_skip_past_delimit__works),
cmocka_unit_test(test__f_fss_skip_past_space__works),
// f_fss_is_space() doesn't use parameter checking.
// f_fss_is_zero_width() doesn't use parameter checking.
cmocka_unit_test(test__f_fss_seek_to_eol__parameter_checking),
+ cmocka_unit_test(test__f_fss_skip_past_combining__parameter_checking),
cmocka_unit_test(test__f_fss_skip_past_delimit__parameter_checking),
cmocka_unit_test(test__f_fss_skip_past_space__parameter_checking),
#include "test-fss-simple_packets_destroy_callback.h"
#include "test-fss-simple_packetss_delete_callback.h"
#include "test-fss-simple_packetss_destroy_callback.h"
+#include "test-fss-skip_past_combining.h"
#include "test-fss-skip_past_delimit.h"
#include "test-fss-skip_past_space.h"
}
else {
if (range->start + width > range->stop) return F_status_set_error(F_complete_not_utf_stop);
- if (range->start + width >= buffer.used) return F_status_set_error(F_complete_not_utf_eos);
+ if (range->start + width > buffer.used) return F_status_set_error(F_complete_not_utf_eos);
if (width == seek_width) {
f_utf_char_t character = 0;
}
else {
if (range->start + width > range->stop) return F_status_set_error(F_complete_not_utf_stop);
- if (range->start + width >= buffer.used) return F_status_set_error(F_complete_not_utf_eos);
+ if (range->start + width > buffer.used) return F_status_set_error(F_complete_not_utf_eos);
}
range->start += width;
}
else {
if (range->start + width > range->stop) return F_status_set_error(F_complete_not_utf_stop);
- if (range->start + width >= buffer.used) return F_status_set_error(F_complete_not_utf_eos);
+ if (range->start + width > buffer.used) return F_status_set_error(F_complete_not_utf_eos);
}
range->start += width;
}
else {
if (range->start + width > range->stop) return F_status_set_error(F_complete_not_utf_stop);
- if (range->start + width >= buffer.used) return F_status_set_error(F_complete_not_utf_eos);
+ if (range->start + width > buffer.used) return F_status_set_error(F_complete_not_utf_eos);
if (width == seek_width) {
f_utf_char_t character = 0;
extern "C" {
#endif
-#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_is_combining_)
+#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_is_combining_) || !defined(_di_f_utf_is_combining_joiner_)
f_status_t private_f_utf_character_is_combining(const f_utf_char_t sequence) {
if (macro_f_utf_char_t_width_is(sequence) == 2) {
return F_false;
}
-#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_is_combining_)
+#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_is_combining_) || !defined(_di_f_utf_is_combining_joiner_)
#ifdef __cplusplus
} // extern "C"
*
* @see f_utf_character_is_alphabetic()
* @see f_utf_character_is_combining()
+ * @see f_utf_character_is_combining_joiner()
* @see f_utf_is_combining()
+ * @see f_utf_is_combining_joiner()
*/
-#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_is_combining_)
+#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_is_combining_) || !defined(_di_f_utf_is_combining_joiner_)
extern f_status_t private_f_utf_character_is_combining(const f_utf_char_t sequence) F_attribute_visibility_internal_d;
-#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_is_combining_)
+#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_combining_) || !defined(_di_f_utf_character_is_combining_joiner_)|| !defined(_di_f_utf_is_combining_) || !defined(_di_f_utf_is_combining_joiner_)
#ifdef __cplusplus
} // extern "C"
extern "C" {
#endif
-#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_)
+#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_combining_joiner_) || !defined(_di_f_utf_is_joiner_)
f_status_t private_f_utf_character_is_joiner(const f_utf_char_t sequence) {
if (macro_f_utf_char_t_width_is(sequence) == 3) {
return F_false;
}
-#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_)
+#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_combining_joiner_) || !defined(_di_f_utf_is_joiner_)
#ifdef __cplusplus
} // extern "C"
* @see f_utf_character_is_alphabetic()
* @see f_utf_character_is_alphabetic_digit()
* @see f_utf_character_is_alphabetic_numeric()
+ * @see f_utf_character_is_combining_joiner()
* @see f_utf_character_is_joiner()
* @see f_utf_is_alphabetic()
* @see f_utf_is_alphabetic_digit()
* @see f_utf_is_alphabetic_numeric()
+ * @see f_utf_is_combining_joiner()
* @see f_utf_is_joiner()
*/
-#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_)
+#if !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_)
extern f_status_t private_f_utf_character_is_joiner(const f_utf_char_t sequence) F_attribute_visibility_internal_d;
-#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_)
+#endif // !defined(_di_f_utf_character_is_alphabetic_) || !defined(_di_f_utf_character_is_alphabetic_digit_) || !defined(_di_f_utf_character_is_alphabetic_numeric_) || !defined(_di_f_utf_character_is_combining_joiner_) || !defined(_di_f_utf_character_is_joiner_) || !defined(_di_f_utf_is_alphabetic_) || !defined(_di_f_utf_is_alphabetic_digit_) || !defined(_di_f_utf_is_alphabetic_numeric_) || !defined(_di_f_utf_is_joiner_)
#ifdef __cplusplus
} // extern "C"
if (macro_f_utf_char_t_width_is(sequence) == 3) {
if (macro_f_utf_char_t_to_char_1(sequence) == 0xe2) {
+ // General Punctuation: U+200D.
+ if (sequence == 0xe2808d00) {
+ return F_true;
+ }
+
// General Punctuation: U+2028 to U+2029.
if (sequence >= 0xf090b4a3 && sequence <= 0xf090b4a7) {
return F_true;
}
+ // General Punctuation: U+2060.
+ if (sequence == 0xe281a000) {
+ return F_true;
+ }
+
// General Punctuation: U+2066 to U+2069.
if (sequence >= 0xe281a600 && sequence <= 0xe281a900) {
return F_true;
return F_okay_stop;
}
else if (range->start + width >= buffer.used) {
- if (width > 1 && range->start + width >= buffer.used + 1) {
+ if (width > 1 && range->start + width > buffer.used) {
return F_status_set_error(F_complete_not_utf_eos);
}
}
#endif // _di_f_utf_is_combining_
+#ifndef _di_f_utf_is_combining_joiner_
+ f_status_t f_utf_is_combining_joiner(const f_string_t sequence, const f_number_unsigned_t width_max) {
+
+ if (width_max < 1) return F_data_not;
+
+ if (macro_f_utf_byte_width_is(*sequence)) {
+ if (macro_f_utf_byte_width_is(*sequence) > width_max) return F_status_set_error(F_complete_not_utf);
+ if (macro_f_utf_byte_width_is(*sequence) == 1) return F_status_set_error(F_utf_fragment);
+
+ f_utf_char_t utf = 0;
+
+ {
+ const f_status_t status = private_f_utf_char_to_character(sequence, width_max, &utf);
+ if (F_status_is_error(status)) return status;
+ }
+
+ if (private_f_utf_character_is_combining(utf)) return F_true;
+
+ return private_f_utf_character_is_joiner(utf);
+ }
+
+ // There are no ASCII combining characters.
+ return F_false;
+ }
+#endif // _di_f_utf_is_combining_joiner_
+
#ifndef _di_f_utf_is_control_
f_status_t f_utf_is_control(const f_string_t sequence, const f_number_unsigned_t width_max) {
// Control and combining characters are zero-width.
if (private_f_utf_character_is_control(utf)) return F_true;
if (private_f_utf_character_is_combining(utf)) return F_true;
+ if (private_f_utf_character_is_joiner(utf)) return F_true;
return private_f_utf_character_is_zero_width(utf);
}
extern f_status_t f_utf_is_combining(const f_string_t sequence, const f_number_unsigned_t width_max);
#endif // _di_f_utf_is_combining_
+/**
+ * Check to see if the entire byte block of the character is a UTF-8 combining character or a UTF-8 joiner character.
+ *
+ * @param sequence
+ * The byte sequence to validate as a character.
+ * There must be enough space allocated to compare against, as limited by width_max.
+ * @param width_max
+ * The maximum width available for checking.
+ * Can be anything greater than 0.
+ *
+ * @return
+ * F_true if a UTF-8 combining character.
+ * F_false if not a UTF-8 combining character.
+ * F_data_not if width_max is less than 1.
+ *
+ * F_complete_not_utf (with error bit set) if character is an incomplete UTF-8 sequence.
+ * F_utf_fragment (with error bit) if character is a UTF-8 fragment.
+ * F_utf_not (with error bit) if Unicode is an invalid Unicode character.
+ */
+#ifndef _di_f_utf_is_combining_joiner_
+ extern f_status_t f_utf_is_combining_joiner(const f_string_t sequence, const f_number_unsigned_t width_max);
+#endif // _di_f_utf_is_combining_joiner_
+
/**
* Check to see if the entire byte block of the character is an ASCII or UTF-8 control character.
*
}
#endif // _di_f_utf_character_is_combining_
+#ifndef _di_f_utf_character_is_combining_joiner_
+ f_status_t f_utf_character_is_combining_joiner(const f_utf_char_t sequence) {
+
+ if (macro_f_utf_char_t_width_is(sequence)) {
+ if (macro_f_utf_char_t_width_is(sequence) == 1) return F_status_set_error(F_utf_fragment);
+
+ if (private_f_utf_character_is_combining(sequence)) return F_true;
+
+ return private_f_utf_character_is_joiner(sequence);
+ }
+
+ // There are no combining characters in ASCII.
+ return F_false;
+ }
+#endif // _di_f_utf_character_is_combining_joiner_
+
#ifndef _di_f_utf_character_is_control_
f_status_t f_utf_character_is_control(const f_utf_char_t sequence) {
extern f_status_t f_utf_character_is_combining(const f_utf_char_t sequence);
#endif // _di_f_utf_character_is_combining_
+/**
+ * Check to see if the entire byte block of the character is a UTF-8 combining character or a UTF-8 joiner character.
+ *
+ * @param sequence
+ * The byte sequence to validate as a character.
+ *
+ * @return
+ * F_true if a UTF-8 combining character.
+ * F_false if not a UTF-8 combining character.
+ *
+ * F_utf_fragment (with error bit) if character is a UTF-8 fragment.
+ * F_utf_not (with error bit) if unicode is an invalid Unicode character.
+ */
+#ifndef _di_f_utf_character_is_combining_joiner_
+ extern f_status_t f_utf_character_is_combining_joiner(const f_utf_char_t sequence);
+#endif // _di_f_utf_character_is_combining_joiner_
+
/**
* Check to see if the entire byte block of the character is an ASCII or UTF-8 control character.
*
build_sources_program test-utf-character_is_alphabetic_digit.c test-utf-is_alphabetic_digit.c
build_sources_program test-utf-character_is_alphabetic_numeric.c test-utf-is_alphabetic_numeric.c
build_sources_program test-utf-character_is_combining.c test-utf-is_combining.c
+build_sources_program test-utf-character_is_combining_joiner.c test-utf-is_combining_joiner.c
build_sources_program test-utf-character_is_control.c test-utf-is_control.c
build_sources_program test-utf-character_is_control_picture.c test-utf-is_control_picture.c
build_sources_program test-utf-character_is_digit.c test-utf-is_digit.c
return fopen("./data/tests/bytesequences/combining-all.txt", "r");
}
+FILE *data__bytesequence_file_open__combining_joiner(void) {
+
+ return fopen("./data/tests/bytesequences/combining_joiner-all.txt", "r");
+}
+
FILE *data__bytesequence_file_open__control(void) {
return fopen("./data/tests/bytesequences/control-all.txt", "r");
*/
extern FILE *data__bytesequence_file_open__combining(void);
+/**
+ * Open the "combining" or "joiner" bytesequence file.
+ *
+ * This assumes the following:
+ * - The file path is relative to the current working directory (tests are run from project root).
+ * - The file path is "data/tests/bytesequences/combining_joiner-all.txt".
+ *
+ * @return
+ * Non-zero on success.
+ * 0 on failure.
+ *
+ * @see fopen()
+ */
+extern FILE *data__bytesequence_file_open__combining_joiner(void);
+
/**
* Open the "control" bytesequence file.
*
--- /dev/null
+#include "test-utf.h"
+#include "test-utf-character_is_combining_joiner.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+void test__f_utf_character_is_combining_joiner__works(void **state) {
+
+ {
+ FILE *file_combining = data__bytesequence_file_open__combining();
+ FILE *file_joiner = data__bytesequence_file_open__joiner();
+
+ assert_non_null(file_combining);
+ assert_non_null(file_joiner);
+
+ f_utf_char_t sequence = 0;
+ ssize_t bytes = 0;
+ long long code = 0;
+
+ f_number_unsigned_t line = 0;
+
+ do {
+ bytes = data__bytesequence_get_line(file_combining, &sequence, &code);
+
+ if (bytes > 0) {
+ const f_status_t status = f_utf_character_is_combining_joiner(sequence);
+
+ if (status != F_true) {
+ printf("[ -------> ] --- [0] Failure at line=%lu sequence='%0x' (%llu).\n", line + 1, sequence, code);
+ }
+
+ assert_int_equal(status, F_true);
+ }
+
+ ++line;
+
+ } while (bytes > 0);
+
+ line = 0;
+
+ do {
+ bytes = data__bytesequence_get_line(file_joiner, &sequence, &code);
+
+ if (bytes > 0) {
+ const f_status_t status = f_utf_character_is_combining_joiner(sequence);
+
+ if (status != F_true) {
+ printf("[ -------> ] --- [1] Failure at line=%lu sequence='%0x' (%llu).\n", line + 1, sequence, code);
+ }
+
+ assert_int_equal(status, F_true);
+ }
+
+ ++line;
+
+ } while (bytes > 0);
+
+ fclose(file_combining);
+ fclose(file_joiner);
+ }
+}
+
+#ifdef __cplusplus
+} // extern "C"
+#endif
--- /dev/null
+/**
+ * FLL - Level 0
+ *
+ * Project: UTF
+ * API Version: 0.8
+ * Licenses: lgpl-2.1-or-later
+ *
+ * Test the function in the utf project.
+ */
+#ifndef _TEST__F_utf_character_is_combining_joiner_h
+#define _TEST__F_utf_character_is_combining_joiner_h
+
+/**
+ * Test that the function works.
+ *
+ * @see f_utf_character_is_combining_joiner()
+ */
+extern void test__f_utf_character_is_combining_joiner__works(void **state);
+
+#endif // _TEST__F_utf_character_is_combining_joiner_h
--- /dev/null
+#include "test-utf.h"
+#include "test-utf-is_combining_joiner.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+void test__f_utf_is_combining_joiner__works(void **state) {
+
+ {
+ FILE *file_combining = data__bytesequence_file_open__combining();
+ FILE *file_joiner = data__bytesequence_file_open__joiner();
+
+ assert_non_null(file_combining);
+ assert_non_null(file_joiner);
+
+ f_utf_char_t sequence = 0;
+ ssize_t bytes = 0;
+ long long code = 0;
+
+ f_number_unsigned_t line = 0;
+
+ do {
+ bytes = data__bytesequence_get_line(file_combining, &sequence, &code);
+
+ if (bytes > 0) {
+ const uint8_t width = macro_f_utf_char_t_width(sequence);
+ char buffer[5] = { 0, 0, 0, 0, 0 };
+
+ buffer[0] = macro_f_utf_char_t_to_char_1(sequence);
+
+ if (width > 1) {
+ buffer[1] = macro_f_utf_char_t_to_char_2(sequence);
+
+ if (width > 2) {
+ buffer[2] = macro_f_utf_char_t_to_char_3(sequence);
+
+ if (width > 3) {
+ buffer[3] = macro_f_utf_char_t_to_char_4(sequence);
+ }
+ }
+ }
+
+ const f_status_t status = f_utf_is_combining_joiner(buffer, 5);
+
+ if (status != F_true) {
+ printf("[ -------> ] --- [0] Failure at line=%lu sequence='%0x' (%llu).\n", line + 1, sequence, code);
+ }
+
+ assert_int_equal(status, F_true);
+ }
+
+ ++line;
+
+ } while (bytes > 0);
+
+ line = 0;
+
+ do {
+ bytes = data__bytesequence_get_line(file_joiner, &sequence, &code);
+
+ if (bytes > 0) {
+ const uint8_t width = macro_f_utf_char_t_width(sequence);
+ char buffer[5] = { 0, 0, 0, 0, 0 };
+
+ buffer[0] = macro_f_utf_char_t_to_char_1(sequence);
+
+ if (width > 1) {
+ buffer[1] = macro_f_utf_char_t_to_char_2(sequence);
+
+ if (width > 2) {
+ buffer[2] = macro_f_utf_char_t_to_char_3(sequence);
+
+ if (width > 3) {
+ buffer[3] = macro_f_utf_char_t_to_char_4(sequence);
+ }
+ }
+ }
+
+ const f_status_t status = f_utf_is_combining_joiner(buffer, 5);
+
+ if (status != F_true) {
+ printf("[ -------> ] --- [1] Failure at line=%lu sequence='%0x' (%llu).\n", line + 1, sequence, code);
+ }
+
+ assert_int_equal(status, F_true);
+ }
+
+ ++line;
+
+ } while (bytes > 0);
+
+ fclose(file_combining);
+ fclose(file_joiner);
+ }
+}
+
+#ifdef __cplusplus
+} // extern "C"
+#endif
--- /dev/null
+/**
+ * FLL - Level 0
+ *
+ * Project: UTF
+ * API Version: 0.8
+ * Licenses: lgpl-2.1-or-later
+ *
+ * Test the function in the utf project.
+ */
+#ifndef _TEST__F_utf_is_combining_joiner_h
+#define _TEST__F_utf_is_combining_joiner_h
+
+/**
+ * Test that the function works.
+ *
+ * @see f_utf_is_combining_joiner()
+ */
+extern void test__f_utf_is_combining_joiner__works(void **state);
+
+#endif // _TEST__F_utf_is_combining_joiner_h
cmocka_unit_test(test__f_utf_character_is_alphabetic_digit__works),
cmocka_unit_test(test__f_utf_character_is_alphabetic_numeric__works),
cmocka_unit_test(test__f_utf_character_is_combining__works),
+ cmocka_unit_test(test__f_utf_character_is_combining_joiner__works),
cmocka_unit_test(test__f_utf_character_is_control__works),
cmocka_unit_test(test__f_utf_character_is_control_picture__works),
cmocka_unit_test(test__f_utf_character_is_digit__works),
cmocka_unit_test(test__f_utf_is_alphabetic_digit__works),
cmocka_unit_test(test__f_utf_is_alphabetic_numeric__works),
cmocka_unit_test(test__f_utf_is_combining__works),
+ cmocka_unit_test(test__f_utf_is_combining_joiner__works),
cmocka_unit_test(test__f_utf_is_control__works),
cmocka_unit_test(test__f_utf_is_control_picture__works),
cmocka_unit_test(test__f_utf_is_digit__works),
#include "test-utf-character_is_alphabetic_digit.h"
#include "test-utf-character_is_alphabetic_numeric.h"
#include "test-utf-character_is_combining.h"
+#include "test-utf-character_is_combining_joiner.h"
#include "test-utf-character_is_control.h"
#include "test-utf-character_is_control_picture.h"
#include "test-utf-character_is_digit.h"
#include "test-utf-is_alphabetic_digit.h"
#include "test-utf-is_alphabetic_numeric.h"
#include "test-utf-is_combining.h"
+#include "test-utf-is_combining_joiner.h"
#include "test-utf-is_control.h"
#include "test-utf-is_control_picture.h"
#include "test-utf-is_digit.h"
#endif // _di_level_1_parameter_checking_
f_fss_skip_past_space(buffer, range, state);
- if (F_status_is_error(state->status)) return;
- if (state->status == F_data_not) return;
+ if (F_status_is_error(state->status) || state->status == F_data_not) return;
if (state->status == F_okay_eol) {
++range->start;
*
* This will record where delimit placeholders exist but will not apply the delimits.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param buffer
* The buffer to read from.
* @param range
*
* The destination string may have NULLs.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param content
* The string to write as (does not stop at NULLS, they are ignored and not written).
* @param complete
*
* This will record where delimit placeholders exist but will not apply the delimits.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param buffer
* The buffer to read from.
* @param range
*
* The destination string may have NULLs.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param object
* The string to write as (does not stop at NULLS, they are ignored and not written).
* @param quote
*
* This will record where delimit placeholders exist but will not apply the delimits.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param buffer
* The buffer to read from.
* @param range
*
* The destination string may have NULLs.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param content
* The string to write as (does not stop at NULLS, they are ignored and not written).
* @param complete
*
* This will record where delimit placeholders exist but will not apply the delimits.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param buffer
* The buffer to read from.
* @param range
*
* The destination string may have NULLs.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param object
* The string to write as (does not stop at NULLS, they are ignored and not written).
* @param complete
* Therefore the object for this content will also need to be found.array[0].array[found.array[0].used].object.
* The object at depth 0 will not be populated, but all nested objects will be populated at their respective depth along with the content.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param buffer
* The buffer to read from.
* @param range
*
* The destination string may have NULLs.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param content
* The string to write as (does not stop at NULLS, they are ignored and not written).
* @param complete
*
* This will record where delimit placeholders exist but will not apply the delimits.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param buffer
* The buffer to read from.
* @param range
*
* The destination string may have NULLs.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param object
* The string to write as (does not stop at NULLS, they are ignored and not written).
* @param complete
#endif // _di_level_1_parameter_checking_
f_fss_skip_past_space(buffer, range, state);
- if (F_status_is_error(state->status)) return;
- if (state->status == F_data_not) return;
+ if (F_status_is_error(state->status) || state->status == F_data_not) return;
if (state->status == F_okay_eol) {
++range->start;
*
* This will record where delimit placeholders exist but will not apply the delimits.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param buffer
* The buffer to read from.
* @param range
*
* The destination string may have NULLs.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param content
* The string to write as (does not stop at NULLS, they are ignored and not written).
* @param quote
*
* This will record where delimit placeholders exist but will not apply the delimits.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param buffer
* The buffer to read from.
* @param range
*
* The destination string may have NULLs.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param object
* The string to write as (does not stop at NULLS, they are ignored and not written).
* @param quote
const f_number_unsigned_t destination_used = destination->used;
- uint8_t is_comment = F_false;
- uint8_t ends_on_eol = F_false;
- uint8_t has_graph = F_false;
- uint8_t do_prepend = prepend ? F_true : F_false;
+ #define ___fl_flag_is_comment___ 0x1
+ #define ___fl_flag_has_graph___ 0x2
+ #define ___fl_flag_is_comment_graph___ 0x3
+ #define ___fl_flag_ends_on_eol___ 0x4
+ #define ___fl_flag_do_prepend___ 0x8
+ #define ___fl_flag_do_prepend_ends_on_eol___ 0xc
+ #define ___fl_flag_graph_combine___ 0x10 // Is a graph only due to combining character.
+ #define ___fl_flag_graph_joiner___ 0x20 // Is a graph only due to a joining character.
+ #define ___fl_flag_graph_joiner_combine___ 0x30
+
+ uint8_t flag = prepend ? ___fl_flag_do_prepend___ : 0x0;
f_number_unsigned_t r = 0;
f_number_unsigned_t start = 0;
if (F_status_set_fine(state->status) == F_interrupt) break;
}
- if (content.string[range->start] == f_fss_slash_s.string[0] && !is_comment) {
+ if (content.string[range->start] == f_fss_slash_s.string[0] && !(flag & ___fl_flag_is_comment___)) {
- if (do_prepend) {
+ if (flag & ___fl_flag_do_prepend___) {
state->status = f_memory_array_increase_by(prepend->used, sizeof(f_char_t), (void **) &destination->string, &destination->used, &destination->size);
if (F_status_is_error(state->status)) break;
state->status = f_string_dynamic_append(*prepend, destination);
if (F_status_is_error(state->status)) break;
- do_prepend = F_false;
+ flag &= ~___fl_flag_do_prepend___;
}
else {
state->status = f_memory_array_increase(state->step_large, sizeof(f_char_t), (void **) &destination->string, &destination->used, &destination->size);
f_fss_skip_past_space(content, range, state);
if (F_status_is_error(state->status)) break;
- if (has_graph) {
+ if (flag & ___fl_flag_has_graph___) {
// Do nothing.
}
else if (content.string[range->start] == f_fss_eol_s.string[0] || range->start >= content.used || range->start > range->stop) {
if (F_status_is_error(state->status)) break;
if (content.string[range->start] != f_fss_eol_s.string[0]) {
- has_graph = F_true;
+ flag |= ___fl_flag_has_graph___;
}
continue;
if (content.string[range->start] == f_fss_eol_s.string[0]) {
destination->string[destination->used++] = f_fss_eol_s.string[0];
- ends_on_eol = F_true;
+ flag |= ___fl_flag_ends_on_eol___;
}
else {
- ends_on_eol = F_false;
+ flag &= ~___fl_flag_ends_on_eol___;
}
range->start = start + 1;
continue;
}
}
- else if (!has_graph && content.string[range->start] == f_fss_extended_list_close_s.string[0] && !is_comment) {
+ else if (!(flag & ___fl_flag_is_comment_graph___) && content.string[range->start] == f_fss_extended_list_close_s.string[0]) {
start = range->start++;
- if (do_prepend) {
+ if (flag & ___fl_flag_do_prepend___) {
state->status = f_string_dynamic_append(*prepend, destination);
if (F_status_is_error(state->status)) break;
- do_prepend = F_false;
+ flag &= ~___fl_flag_do_prepend___;
}
- has_graph = F_true;
+ flag |= ___fl_flag_has_graph___;
f_fss_skip_past_space(content, range, state);
if (F_status_is_error(state->status)) break;
if (content.string[range->start] == f_fss_eol_s.string[0] || range->start >= content.used || range->start > range->stop) {
if (content.string[range->start] == f_fss_eol_s.string[0] && prepend) {
- do_prepend = F_true;
- ends_on_eol = F_true;
+ flag |= ___fl_flag_do_prepend_ends_on_eol___;
}
else {
- ends_on_eol = F_false;
+ flag &= ~___fl_flag_ends_on_eol___;
}
if (ignore && ignore->used) {
continue;
}
- else if (content.string[range->start] == f_fss_comment_s.string[0] && !has_graph) {
- is_comment = F_true;
+ else if (content.string[range->start] == f_fss_comment_s.string[0] && !(flag & ___fl_flag_has_graph___)) {
+ flag |= ___fl_flag_is_comment___;
}
else if (content.string[range->start] == f_fss_eol_s.string[0]) {
- has_graph = F_false;
- is_comment = F_false;
+ flag &= ~(flag & ___fl_flag_is_comment_graph___);
}
- else if (f_fss_is_graph(content, *range, state) == F_false) {
- if (F_status_is_error(state->status)) break;
+ else if (f_fss_is_graph(content, *range, state) == F_true) {
+ flag |= ___fl_flag_has_graph___;
}
else {
- has_graph = F_true;
+ if (F_status_is_error(state->status)) break;
+
+ // TODO: flag |= ___fl_flag_graph_combine___;
+ // TODO: flag |= ___fl_flag_graph_joiner___;
}
if (content.string[range->start] != f_fss_placeholder_s.string[0]) {
- if (do_prepend) {
+ if (flag & ___fl_flag_do_prepend___) {
state->status = f_string_dynamic_append(*prepend, destination);
if (F_status_is_error(state->status)) break;
- do_prepend = F_false;
+ flag &= ~___fl_flag_do_prepend___;
}
if (content.string[range->start] == f_fss_eol_s.string[0] && prepend) {
- do_prepend = F_true;
- ends_on_eol = F_true;
+ flag |= ___fl_flag_do_prepend_ends_on_eol___;
}
else {
- ends_on_eol = F_false;
+ flag &= ~___fl_flag_ends_on_eol___;
}
width = macro_f_utf_byte_width(content.string[range->start]);
+ // ___fl_flag_graph_combine___ and ___fl_flag_graph_joiner___ are only set if the characters can be found, so it is therefore safe to trust that range->start + width can be used.
+ if (flag & ___fl_flag_graph_combine___) {
+ width += macro_f_utf_byte_width(content.string[range->start + width]);
+ }
+
state->status = f_memory_array_increase_by(width, sizeof(f_char_t), (void **) &destination->string, &destination->used, &destination->size);
if (F_status_is_error(state->status)) break;
memcpy(destination->string + destination->used, content.string + range->start, width);
destination->used += width;
+
+ if (flag & ___fl_flag_graph_combine___) {
+ state->status = f_utf_buffer_increment(content, range, 2);
+ if (F_status_is_error(state->status)) break;
+
+ flag &= ~___fl_flag_graph_combine___;
+
+ continue;
+ }
}
state->status = f_utf_buffer_increment(content, range, 1);
if (F_status_is_error(state->status)) break;
+
} // while
if (F_status_is_error(state->status)) {
return;
}
- if (!ends_on_eol) {
+ if (!(flag & ___fl_flag_ends_on_eol___)) {
destination->string[destination->used++] = f_fss_eol_s.string[0];
}
else {
state->status = F_okay;
}
+
+ #undef ___fl_flag_is_comment___
+ #undef ___fl_flag_has_graph___
+ #undef ___fl_flag_is_comment_graph___
+ #undef ___fl_flag_ends_on_eol___
+ #undef ___fl_flag_do_prepend___
+ #undef ___fl_flag_do_prepend_ends_on_eol___
+ #undef ___fl_flag_graph_combine___
+ #undef ___fl_flag_graph_joiner___
+ #undef ___fl_flag_graph_joiner_combine___
}
#endif // _di_fl_fss_extended_list_content_write_
*
* This does not utilize recursion and has a max content depth of 1, see fss-0008 (Embedded List) for that purpose.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param buffer
* The buffer to read from.
* @param range
*
* The destination string may have NULLs.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param content
* The string to write as (does not stop at NULLS, they are ignored and not written).
* @param complete
*
* This will record where delimit placeholders exist but will not apply the delimits.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param buffer
* The buffer to read from.
* @param range
*
* The destination string may have NULLs.
*
+ * Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+ *
* @param object
* The string to write as (does not stop at NULLS, they are ignored and not written).
* @param complete
void private_fl_fss_basic_or_extended_read(const f_string_static_t buffer, const uint8_t flag, f_range_t * const range, f_range_t * const found, uint8_t * const quote, f_number_unsigneds_t * const delimits, f_state_t * const state) {
f_fss_skip_past_space(buffer, range, state);
-
- if (F_status_is_error(state->status) || state->status == F_data_not) {
- if (F_status_set_fine(state->status) == F_complete_not_utf_start) {
- f_fss_skip_past_combining(buffer, range, state);
- if (F_status_is_error(state->status) || state->status == F_data_not) return;
- }
- else return;
- }
+ if (F_status_is_error(state->status) || state->status == F_data_not) return;
if (state->status == F_okay_eol) {
// Move the start position to after the EOL.
}
// Handle quote support.
- f_char_t quote_found = 0;
+ f_char_t quote_found = 0; // TODO: check if quote has combining characters?
if (quote) *quote = f_fss_quote_type_none_e;
f_number_unsigned_t first_slash = 0;
f_number_unsigned_t slash_count = 0;
f_number_unsigned_t location = 0;
+ f_number_unsigned_t previous = 0;
while (range->start <= range->stop && range->start < buffer.used) {
}
} // while
- if (f_fss_is_graph(buffer, *range, state) == F_false) {
+ if (f_fss_is_graph(buffer, *range, state) != F_true) {
if (F_status_is_error(state->status)) return;
+
+ // Anything other than a new line must be checked if a combining or joiner is in the next character.
+ if (buffer.string[range->start] != f_fss_eol_s.string[0] && range->start <= range->stop && range->start < buffer.used) {
+ previous = range->start;
+
+ state->status = f_utf_buffer_increment(buffer, range, 1);
+ if (F_status_is_error(state->status)) return;
+
+ state->status = f_utf_is_combining_joiner(
+ buffer.string + range->start,
+ (range->stop - range->start) + 1 > buffer.used - range->start
+ ? buffer.used - range->start
+ : (range->stop - range->start) + 1
+ );
+
+ // When status is F_true, then this is a combining or joiner that causes the previous character to be treated as a graph.
+ if (state->status == F_false) {
+ range->start = previous;
+ } else if (F_status_is_error(state->status)) {
+ return;
+ }
+ }
}
- else {
+
+ if (state->status == F_true) {
while (range->start <= range->stop && range->start < buffer.used && buffer.string[range->start] != f_fss_eol_s.string[0]) {
if (state->interrupt) {
void private_fl_fss_basic_write(const uint8_t flag, const f_string_static_t object, const uint8_t quote, f_range_t * const range, f_string_dynamic_t * const destination, f_state_t * const state, void * const internal) {
f_fss_skip_past_space(object, range, state);
-
- if (F_status_is_error(state->status) || state->status == F_data_not) {
- if (F_status_set_fine(state->status) == F_complete_not_utf_start) {
- f_fss_skip_past_combining(object, range, state);
- if (F_status_is_error(state->status) || state->status == F_data_not) return;
- }
- else return;
- }
+ if (F_status_is_error(state->status) || state->status == F_data_not) return;
if (state->status == F_okay_eos) {
state->status = F_data_not_eos;
*
* Errors (with error bit) from: f_memory_array_increase().
* Errors (with error bit) from: f_memory_array_increase_by().
+ * Errors (with error bit) from: f_fss_is_combining().
* Errors (with error bit) from: f_fss_is_graph().
* Errors (with error bit) from: f_fss_is_space().
* Errors (with error bit) from: f_fss_is_zero_width().
* Errors (with error bit) from: f_fss_skip_past_delimit().
* Errors (with error bit) from: f_fss_skip_past_space().
* Errors (with error bit) from: f_utf_buffer_increment().
+ * Errors (with error bit) from: f_utf_is_combining_joiner().
*
* @see f_memory_array_increase()
* @see f_memory_array_increase_by()
+ * @see f_fss_is_combining()
* @see f_fss_is_graph()
* @see f_fss_is_space()
* @see f_fss_is_zero_width()
* @see f_fss_skip_past_delimit()
* @see f_fss_skip_past_space()
* @see f_utf_buffer_increment()
+ * @see f_utf_is_combining_joiner()
* @see fl_fss_basic_object_read()
* @see fl_fss_extended_object_read()
* @see fl_fss_extended_content_read()
.PP
This will record where delimit placeholders exist but will not apply the delimits.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
The destination string may have NULLs.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt.
.PP
Must not be NULL.
.PP
This will record where delimit placeholders exist but will not apply the delimits.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
The destination string may have NULLs.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
This will record where delimit placeholders exist but will not apply the delimits.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
The destination string may have NULLs.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
This will record where delimit placeholders exist but will not apply the delimits.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
The destination string may have NULLs.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
This operates on the assumption that found.array[0].array[found.array[0].used].content is the current content being processed. Therefore the object for this content will also need to be found.array[0].array[found.array[0].used].object. The object at depth 0 will not be populated, but all nested objects will be populated at their respective depth along with the content.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
The state.data must be of type fl_fss_data_embedded_list_t with all properties as valid.
.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
The destination string may have NULLs.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
This will record where delimit placeholders exist but will not apply the delimits.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
The destination string may have NULLs.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
This will record where delimit placeholders exist but will not apply the delimits.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
f_fss_state_quote_not_e: Explicitly disable quotes, treating quotes and their respective delimits as normal characters.
.PP
The destination string may have NULLs.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
This does not utilize recursion and has a max content depth of 1, see fss-0008 (Embedded List) for that purpose.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
The destination string may have NULLs.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
This will record where delimit placeholders exist but will not apply the delimits.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
The destination string may have NULLs.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
This will record where delimit placeholders exist but will not apply the delimits.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
.PP
The destination string may have NULLs.
.PP
+Combining spaces at the start must be handled by the caller to avoid F_complete_not_utf_start errors relating to combining characters.
+.PP
When state.interrupt() returns, only F_interrupt and F_interrupt_not are processed. Error bit designates an error but must be passed along with F_interrupt. All other statuses are ignored.
.PP
Must not be NULL.
# fss-0002 iki-0000
#
# license open-standard-license-1.0-or-later
-# version 2026/05/31
+# version 2026/06/20
#
# This file (assumed to be named fss.txt) can be more easily read using the following iki_read commands:
# iki_read fss.txt +Q -w -rrrrrrrr anti-KISS anti-KISS ASCII ASCII BOM BOM FSS FSS KISS KISS UTF-8 UTF-8 URL URL XML XML -WWW character "'" "'" code '"' '"' italic '"' '"'
These specifications refer to characters that have no printable representation as italic:"non-printable".
White spaces characters that are printable, such as tabs and spaces, must be considered the same type for the purposes of parsing.
Non-printing white spaces characters (zero-width characters) are ignored, are treated as placeholders for processing with the exception of combining characters.
- White spaces that use combining characters result in printable characters and the resulting combination is treated as not white space.
+ Using a combining character automatically causes the preceding character to be considered a graph character.
+ White space characters that use combining characters are treated as a graph character.
+ Zero-width characters that use combining characters are treated as a graph character.
+ The new line character:"\n" (unicode:"U+000A") is an exception to this combining characters rule and is always treated as a new line without regard to any combining character.
A specification may override this handling of combining characters through explicit definitions.
- Zero-width characters that use combining characters are treated as non-printing characters and are skipped.
+ For the purposes of processing, the use of a joiner character (unicode:"U+200D" and unicode:"U+2060") is to be treated in the same way as a combining character.
+ This is considered the case even if the joiner joins two non-graph characters.
In terms of processing, it is recommended that the code:"NULL" character is not considered the end of a string, but this is only a suggestion.
- Any specification may chose to limit, restrict, or otherwise prohibit special Unicode characters such as combining characters or zero-width characters.
+ Any specification may chose to limit, restrict, or otherwise prohibit special Unicode characters such as combining characters, zero-width, and joiner characters.
- The current behavior of existing tools treat new line character:"\n" (unicode:"U+000A") the same regardless if a combining character follows this.
+ The current behavior of common tools treat new line character:"\n" (unicode:"U+000A") the same regardless if a combining character follows this.
As an exception to the above rules (in order to avoid confusion with how existing software handles this behavior) a new line character:"\n" (unicode:"U+000A") is always treated as a such even if followed by a combining character.
In this situation the combining characters at the start can be considered invalid, however the standard behavior is to ignore them and not treat them as part of the data.
Should the data be sensitive in nature or require strict structure, then treating this situation as an error invalid is recommended.
- Should tools, some time in the future, decide to combine onto character:"\n" (unicode:"U+000A"), then this standard can (and should) be reveiwed and updated to a more proper behavior of actually combining onto the control character.
+ Should tools, some time in the future, decide to combine onto character:"\n" (unicode:"U+000A"), then this standard can (and should) be reviewed and potentially updated to a more proper behavior of actually combining onto the control character.
Unless otherwise specified, new lines designate the potential start (or end) of an Object or Content.
"Object 1\" is an unterminated object due to the escaped closing quote.
"Object 1\\" has content starting at the 'has', with an Object named "Object 1\".
+ Unless otherwise specified, quotes with combining or joining characters are not to be considered a quote for these purposes.
+
Unless otherwise specified, character/data delimits are performed only when required and not unilaterally.
In the case of Objects, delimits would only apply when that Object could be potentially identified as an Object.
For example, code:"FSS-0001 (Extended)" needs quotes to group parts that include spaces, if there is no initial quote, then a quote following the data italic:"must not" be delimited.
projects / fll / commitdiff