diff options
Diffstat (limited to 'README.txt')
| -rw-r--r-- | README.txt | 163 |
1 files changed, 163 insertions, 0 deletions
diff --git a/README.txt b/README.txt new file mode 100644 index 0000000..9544b49 --- /dev/null +++ b/README.txt @@ -0,0 +1,163 @@ +Reichweit.IT unicode library +============================ + +This software package contains a C++ library for Unicode encoding conversion +and command line tools which apply those functions in example runtime programs: +recode and validate. + + +C++ interface (package libunicode-dev) +-------------------------------------- + +This library includes multiple encoding specification concepts to choose from: +While explicit specification of source and destination encodings are possible, +implicit specification of encoding of Unicode UTF encodings is also implemented +via the respective C++ types: For char8_t, char16_t and char32_t, the +respective UTF-8, UTF-16 and UTF-32 encoding is automatically used. In case of +C++17 where char8_t is not implemented, char is used instead. The same applies +for the std::basic_string<> specializations std::u8string (or std::string on +C++17), std::u16string and std::u32string. + +The main purpose of this library is conversion (and validation) between Unicode +encodings. However, Latin-1 (i.e. ISO 8859-1) and Latin-9 (i.e. ISO 8859-15) +are also implemented for practical reasons. Since the Latin character sets are +also encoded in char and std::string (at least necessarily on C++17), the Latin +encodings must be specified explicitly for disambiguation where Unicode is used +by default otherwise. I.e. UTF-8 is the default for all 8 bit character types, +UTF-16 is the default for 16 bit character types and UTF-32 is the default for +32 bit character types. + +Besides support for different character and string types from the STL, common +container types like std::vector, std::deque, std::list and std::array (the +latter only as source) are supported. + +The basic convention for the conversion interface is: + + to = unicode::convert<FromType, ToType>(from); + +where FromType and ToType can be one of: + +(1) Character type like char, char8_t, char16_t and char32_t +(2) Container type like std::string, std::list<char>, std::deque<char32_t> +(3) Explicit encoding like unicode::UTF_8, unicode::UTF_16, unicode::UTF_32, + unicode::ISO_8859_1 or unicode::ISO_8859_15 + +For the validation interface, the same principle applies: + + bool flag = unicode::is_valid_utf<FromType>(from); + +There is also a Unicode character validation function which operates on Unicode +character values directly, i.e. no specific encoding is used but 32 bit (or +less) values are evaluated for a valid Unicode character: + + bool flag = unicode::is_valid_unicode(character_value); + +While this validates a Unicode value in general, it doesn't tell if the +specified value is actually designated in an actual Unicode version. E.g. as of +2022, in current Unicode version 14.0, the character 0x1FABA "NEST WITH EGGS" +is designated, but not 0x1FABB. Both of them would be detected as "valid" by +unicode::is_valid_unicode(). See also: + +https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt + + +Examples: + +#include <unicode.h> +... + +C++17 conversion of a UTF-8 string to UTF-16: + + std::string utf8_value {u8"äöü"}; + std::u16string utf16_value{unicode::convert<char, char16_t>(utf8_value)}; + +C++20 conversion of a UTF-8 string to UTF-16: + + std::u8string utf8_value {u8"äöü"}; + std::u16string utf16_value{unicode::convert<char8_t, char16_t>(utf8_value)}; + +The following encodings are implicitly deducted from types: + * char resp. char8_t (C++20): UTF-8 + * char16_t: UTF-16 + * char32_t: UTF-32 + +Specification via container types: + + std::deque<char> utf8_value {...}; + std::list<wchar_t> utf16_value{unicode::convert<std::deque<char>, std::list<wchar_t>>(utf8_value)}; + +Explicit encoding specification: + + std::string value {"äöü"}; + std::u32string utf32_value{unicode::convert<unicode::ISO_8859_1, unicode::UTF_32>(value)}; + +Supported encodings are: + + * unicode::UTF_8 + * unicode::UTF_16 + * unicode::UTF_32 + * unicode::ISO_8859_1 + * unicode::ISO_8859_15 + +Supported basic types for source and target characters: + * char + * char8_t (C++20) + * wchar_t (UTF-16 on Windows, UTF-32 on Linux) + * char16_t + * char32_t + * uint8_t, int8_t + * uint16_t, int16_t + * uint32_t, int32_t + * basically, all basic 8-bit, 16-bit and 32-bit that can encode + UTF-8, UTF-16 and UTF-32, respectively. + +Supported container types: + * All std container types that can be iterated (vector, list, deque, array) + * Source and target containers can be different container types + +Validation can be done like this: + + bool valid{unicode::is_valid_utf<char16_t>(utf16_value)}; + +Or via explicit encoding specification: + + bool valid{unicode::is_valid_utf<unicode::UTF_8>(utf8_value)}; + + +CLI interface (package unicode-tools) +------------------------------------- + +* unicode-recode + + Usage: recode <from-format> <from-file> <to-format> <to-file> + Format: + UTF-8 UTF-8 + UTF-16 UTF-16, native endian + UTF-16LE UTF-16, little endian + UTF-16BE UTF-16, big endian + UTF-32 UTF-32, native endian + UTF-32LE UTF-32, little endian + UTF-32BE UTF-32, big endian + ISO-8859-1 ISO-8859-1 (Latin-1) + ISO-8859-15 ISO-8859-15 (Latin-9) + Exit code: 0 if valid, 1 otherwise. + +* unicode-validate + + Usage: validate <format> <file> + Format: + UTF-8 UTF-8 + UTF-16 UTF-16, big or little endian + UTF-16LE UTF-16, little endian + UTF-16BE UTF-16, big endian + UTF-32 UTF-32, big or little endian + UTF-32LE UTF-32, little endian + UTF-32BE UTF-32, big endian + Exit code: 0 if valid, 1 otherwise. + + +Contact +------- + +Reichwein IT <mail@reichwein.it> + |