Special Unicode characters
val u_bom : Stdlib.Uchar.t
u_bom
is the byte order mark (BOM) character (U+FEFF
).
val u_rep : Stdlib.Uchar.t
u_rep
is the replacement character (U+FFFD
).
Unicode encoding schemes
The type for Unicode encoding schemes.
The type for encoding schemes decoded by Uutf
. Unicode encoding schemes plus US-ASCII and ISO/IEC 8859-1 (latin-1).
val encoding_of_string : string -> decoder_encoding option
encoding_of_string s
converts a (case insensitive) IANA character set name to an encoding.
val encoding_to_string : [< decoder_encoding ] -> string
encoding_to_string e
is a IANA character set name for e
.
Decode
The type for input sources. With a `Manual
source the client must provide input with Manual.src
.
The type for newline normalizations. The variant argument is the normalization character.
`ASCII
, normalizes CR (U+000D
), LF (U+000A
) and CRLF (<U+000D
,U+000A
>).`NLF
, normalizes the Unicode newline function (NLF). This is NEL (U+0085
) and the normalizations of`ASCII
.`Readline
, normalizes for a Unicode readline function. This is FF (U+000C
), LS (U+2028
), PS (U+2029
), and the normalizations of`NLF
.
Used with an appropriate normalization character the `NLF
and `Readline
normalizations allow to implement all the different recommendations of Unicode's newline guidelines (section 5.8 in Unicode 9.0.0).
val decoder : ?nln:[< nln ] -> ?encoding:[< decoder_encoding ] -> [< src ] -> decoder
decoder nln encoding src
is a decoder that inputs from src
.
Byte order mark. Byte order mark (BOM) constraints are application dependent and prone to misunderstandings (see the FAQ). Hence, Uutf
decoders have a simple rule: an initial BOM is always removed from the input and not counted in character position tracking. The function decoder_removed_bom
does however return true
if a BOM was removed so that all the information can be recovered if needed.
For UTF-16BE and UTF-16LE the above rule is a violation of conformance D96 and D97 of the standard. Uutf
favors the idea that if there's a BOM, decoding with `UTF_16
or the `UTF_16XX
corresponding to the BOM should decode the same character sequence (this is not the case if you stick to the standard). The client can however regain conformance by consulting the result of decoder_removed_bom
and take appropriate action.
Encoding. encoding
specifies the decoded encoding scheme. If `UTF_16
is used the endianness is determined according to the standard: from a BOM if there is one, `UTF_16BE
otherwise.
If encoding
is unspecified it is guessed. The result of a guess can only be `UTF_8
, `UTF_16BE
or `UTF_16LE
. The heuristic looks at the first three bytes of input (or less if impossible) and takes the first matching byte pattern in the table below.
xx = any byte
.. = any byte or no byte (input too small)
pp = positive byte
uu = valid UTF-8 first byte
Bytes | Guess | Rationale
---------+-----------+-----------------------------------------------
EF BB BF | `UTF_8 | UTF-8 BOM
FE FF .. | `UTF_16BE | UTF-16BE BOM
FF FE .. | `UTF_16LE | UTF-16LE BOM
00 pp .. | `UTF_16BE | ASCII UTF-16BE and U+0000 is often forbidden
pp 00 .. | `UTF_16LE | ASCII UTF-16LE and U+0000 is often forbidden
uu .. .. | `UTF_8 | ASCII UTF-8 or valid UTF-8 first byte.
xx xx .. | `UTF_16BE | Not UTF-8 => UTF-16, no BOM => UTF-16BE
.. .. .. | `UTF_8 | Single malformed UTF-8 byte or no input.
This heuristic is compatible both with BOM based recognitition and JSON-like encoding recognition that relies on ASCII being present at the beginning of the stream. Also, decoder_removed_bom
will tell the client if the guess was BOM based.
Newline normalization. If nln
is specified, the given newline normalization is performed, see nln
. Otherwise all newlines are returned as found in the input.
Character position. The line number, column number, byte count and character count of the last decoded character (including `Malformed
ones) are respectively returned by decoder_line
, decoder_col
, decoder_byte_count
and decoder_count
. Before the first call to decode
the line number is 1
and the column is 0
. Each decode
returning `Uchar
or `Malformed
increments the column until a newline. On a newline, the line number is incremented and the column set to zero. For example the line is 2
and column 0
after the first newline was decoded. This can be understood as if decode
was moving an insertion point to the right in the data. A newline is anything normalized by `Readline
, see nln
.
Uutf
assumes that each Unicode scalar value has a column width of 1. The same assumption may not be made by the display program (e.g. for emacs
' compilation mode you need to set compilation-error-screen-columns
to nil
). The problem is in general difficult to solve without interaction or convention with the display program's rendering engine. Depending on the context better column increments can be implemented by using Uucp
.Break.tty_width_hint or grapheme cluster boundaries (see Uuseg
).
val decode : decoder -> [ `Await | `Uchar of Stdlib.Uchar.t | `End | `Malformed of string ]
decode d
is:
`Await
ifd
has a`Manual
input source and awaits for more input. The client must useManual.src
to provide it.`Uchar u
if a Unicode scalar valueu
was decoded.`End
if the end of input was reached.`Malformed bytes
if thebytes
sequence is malformed according to the decoded encoding scheme. If you are interested in a best-effort decoding you can still continue to decode after an error until the decoder synchronizes again on valid bytes. It may however be a good idea to signal the malformed characters by adding anu_rep
character to the parsed data, see the examples.
Note. Repeated invocation always eventually returns `End
, even in case of errors.
val decoder_encoding : decoder -> decoder_encoding
decoder_encoding d
is d
's the decoded encoding scheme of d
.
Warning. If the decoder guesses the encoding or uses `UTF_16
, rely on this value only after the first `Uchar
was decoded.
val decoder_line : decoder -> int
decoder_line d
is the line number of the last decoded (or malformed) character. See decoder
for details.
val decoder_col : decoder -> int
decoder_col d
is the column number of the last decoded (or malformed) character. See decoder
for details.
val decoder_byte_count : decoder -> int
decoder_byte_count d
is the number of bytes already decoded on d
(including malformed ones). This is the last decode
's end byte offset counting from the beginning of the stream.
val decoder_count : decoder -> int
decoder_count d
is the number of characters already decoded on d
(including malformed ones). See decoder
for details.
val decoder_removed_bom : decoder -> bool
val pp_decode : Stdlib.Format.formatter -> [< `Await | `Uchar of Stdlib.Uchar.t | `End | `Malformed of string ] -> unit
pp_decode ppf v
prints an unspecified representation of v
on ppf
.
Encode
The type for output destinations. With a `Manual
destination the client must provide output storage with Manual.dst
.
encoder encoding dst
is an encoder for encoding
that outputs to dst
.
Note. No initial BOM is encoded. If needed, this duty is left to the client.
val encode : encoder -> [< `Await | `End | `Uchar of Stdlib.Uchar.t ] -> [ `Ok | `Partial ]
encode e v
is :
`Partial
iffe
has a`Manual
destination and needs more output storage. The client must useManual.dst
to provide a new buffer and then callencode
with`Await
until`Ok
is returned.`Ok
when the encoder is ready to encode a new`Uchar
or`End
For `Manual
destination, encoding `End
always returns `Partial
, the client should continue as usual with `Await
until `Ok
is returned at which point Manual.dst_rem
e
is guaranteed to be the size of the last provided buffer (i.e. nothing was written).
Raises. Invalid_argument
if an `Uchar
or `End
is encoded after a `Partial
encode.
Manual sources and destinations.
module Manual : sig ... end
Manual sources and destinations.
String folders and Buffer encoders
module String : sig ... end
Fold over the characters of UTF encoded OCaml string
values.
module Buffer : sig ... end
UTF encode characters in OCaml Buffer
.t values.