# Extra libraries

Beyond the standard library, an app can offer any of these extra libraries: `json`,
`base64`, `hex`, `url`, `csv`, `decimal`, `stats`, `template`, `bit32`, `hash`, `tpath`,
`validate` and `text`. Which ones your scripts can use is up to the app. They follow the
same rules as the standard library: every function gives the same result for the same
input, none of them reaches files, the network, the clock or randomness, and none can
crash the app. Bad input is an ordinary error you can catch (`invalidArgument`,
`typeMismatch`, `divisionByZero` and so on); running out of a limit (string or table
size, time) can't be caught.

All of them treat strings as **text** (Unicode characters). Functions that deal in bytes
work on the text's UTF-8 bytes, and anything decoded that isn't valid UTF-8 text is
refused rather than producing a broken string.

Every function is **fully typed**, so the checker knows its parameters and results.
Where a result depends on an argument's type, the checker connects them:
`table.clone(t)`, `deepcopy(t)` and `extend(dst, …)`, `assert(v)`, `typecheck(v, …)`,
and `tpath.set(t, …)` and `delete(t, …)` return the same type as their argument.

In the examples, `-->` shows what a line produces.

---

## `json`

Turns script values into JSON text and back, with the same result every time.

- `json.encode(value [, opts])` gives a string.
  - `opts.indent` (1–10) pretty-prints with that many spaces; leaving it out (or 0) gives
    compact output.
  - `opts.sort_keys` (a boolean) sorts object keys. Otherwise keys stay in **the order they
    were added**, matching `pairs`.
  - A table becomes a JSON **array** when its entries are exactly `1…#t`, and an
    **object** otherwise (number keys become strings). An empty table becomes `[]`.
  - Functions, infinite or NaN numbers, object keys that aren't strings or numbers, and
    tables that contain themselves are `invalidArgument` errors, as is nesting deeper than
    200 levels.
- `json.decode(s)` gives a value. Objects become tables with string keys, arrays become
  lists starting at 1, and `null` becomes `nil`, so a `null` field is simply absent.
  Numbers keep their form (`42` is an integer; `42.0` and `1e3` are floating-point).
  `\uXXXX` escapes, including surrogate pairs, are supported. Badly formed input, or
  nesting deeper than 200 levels, is an `invalidArgument` error giving the position.

```lua
local order = json.decode('{"id": 7, "items": [{"sku": "A", "qty": 2}], "note": null}')
order.items[1].sku                          --> "A"
order.note                                  --> nil
json.encode({ b = 1, a = { 1, 2 } })        --> '{"b":1,"a":[1,2]}'
json.encode({ b = 1, a = 2 }, { sort_keys = true })  --> '{"a":2,"b":1}'
json.encode({})                             --> "[]"
json.decode("{oops")                        --> error: at position 2
```

**Keeping JSON exactly.** Standard Lua has no `null`, so by default a `null` field simply
disappears, and an empty table encodes as `[]`. When the difference matters, for
example when the JSON is checked against a schema, use the explicit forms:

- `json.decode(s, { explicit_null = true })` keeps the document exactly: `null` becomes
  `json.null()`, and every array and object remembers which it was, so encoding the
  result gives the same JSON back.
- `json.null()` is JSON's `null` as a value a table can hold, and `json.isnull(v)` tests
  for it. Compare with `json.isnull`, not `==`.
- `json.array([t])` and `json.object([t])` make a table that always encodes as an array
  or an object, even when it's empty.

These are sealed: a table you build yourself can't pass for one.

```lua
local doc = json.decode('{"note": null, "tags": [], "meta": {}}', { explicit_null = true })
json.isnull(doc.note)                       --> true
json.encode(doc, { sort_keys = true })      --> '{"meta":{},"note":null,"tags":[]}'
json.decode('{"note": null}').note          --> nil
json.encode({ items = json.array(), extra = json.object() }, { sort_keys = true })  --> '{"extra":{},"items":[]}'
json.encode({ value = json.null() })        --> '{"value":null}'
json.encode(json.array({ 1, 2 }))           --> "[1,2]"
json.isnull({})                             --> false
```

## `base64`

- `base64.encode(s [, urlsafe])` gives Base64 text (RFC 4648). With `urlsafe = true` it
  uses the `-_` alphabet (still padded with `=`).
- `base64.decode(s [, urlsafe])` gives the decoded text. It accepts either alphabet, and
  refuses a wrong length, stray characters, or a result that isn't valid UTF-8 text.
- `base64.to_hex(s)` gives the lowercase hex of the bytes the Base64 text encodes. It
  accepts either alphabet and works for any bytes, even ones that aren't UTF-8 text
  (such as a signature), because they never pass through a Lua string. It refuses a
  wrong length or stray characters.

```lua
base64.encode("user:secret")               --> "dXNlcjpzZWNyZXQ="
base64.decode("dXNlcjpzZWNyZXQ=")          --> "user:secret"
base64.encode("??>", true)                 --> "Pz8-"
base64.to_hex("/w==")                      --> "ff"   -- base64.decode refuses these bytes
```

## `hex`

- `hex.encode(s)` gives lowercase hexadecimal, two digits per byte.
- `hex.decode(s)` gives the text back, and refuses an odd length, characters that aren't
  hex digits, or a result that isn't valid UTF-8 text.
- `hex.to_base64(s [, urlsafe])` gives the Base64 text of the bytes the hex encodes,
  with `urlsafe` as in `base64.encode`. Like `base64.to_hex` it works for any bytes, so
  it turns a hex digest into Base64; it refuses an odd length or non-hex characters.

```lua
hex.encode("Hi")     --> "4869"
hex.encode("é")      --> "c3a9"
hex.decode("4869")   --> "Hi"
hex.to_base64("ff")  --> "/w=="   -- hex.decode refuses these bytes
```

## `url`

- `url.encode(s)` and `url.decode(s)` percent-encode a single part of a URL (RFC 3986):
  letters, digits and `-._~` pass through, and everything else becomes `%XX` for each
  UTF-8 byte. `decode` leaves `+` as it is.
- `url.encode_query(params)` gives `k=v&…` with **keys sorted**, so the result is always
  the same, and keys and values percent-encoded. Values can be strings, numbers or
  booleans.
- `url.decode_query(s)` gives a table of the decoded pairs. `+` becomes a space (as web
  forms send it), a key with no value maps to `""`, and a later repeat of a key wins.
  Badly formed escapes are refused.

```lua
url.encode("a b&c")                               --> "a%20b%26c"
url.encode_query({ q = "two words", page = 2 })   --> "page=2&q=two%20words"
url.decode_query("a=1&b=two+words&flag")          --> {a = "1", b = "two words", flag = ""}
```

## `csv`

Follows RFC 4180. `opts.delimiter` is a single character (`","` unless you say
otherwise).

- `csv.parse(s [, opts])` gives a list of rows, each a list of text fields. Quoted
  fields, delimiters and line breaks inside quotes, and `""` for a quote are all
  handled. Windows and Unix line endings both end a row, and a final line break doesn't
  add an empty row. Badly formed quoting is read as sensibly as possible rather than
  failing.
- `csv.format(rows [, opts])` gives CSV text. A field containing the delimiter, a quote
  or a line break is quoted, with `""` for each quote, and rows are joined with `"\n"`.
  Fields can be strings, numbers or booleans (`nil` gives an empty field).

```lua
local rows = csv.parse("name,qty\nwidget,3\n\"gizmo, large\",10\n")
#rows          --> 3
rows[3][1]     --> "gizmo, large"
csv.format({ { "name", "qty" }, { "re,sold", 5 } })   --> 'name,qty\n"re,sold",5'
csv.parse("a;b", { delimiter = ";" })                 --> {{"a", "b"}}
```

## `decimal`

Exact decimals, for money and similar values. A decimal can't be changed once made, and
has a value and a scale (the number of decimal places, 0–18): 12.34 has scale 2. Every
operation checks for overflow and division by zero. The library provides the
**`Decimal`** type, so the checker follows decimals through your code
(`decimal.add(a, b)` is a `Decimal`) and refuses tables that aren't decimals.

`Decimal` is a **sealed type** (see the Language Guide, §5), so ordinary operators work:

```lua
local price, qty, shipping = decimal.new("19.99"), 3, decimal.new("4.95")
local total = price * qty + shipping              -- a Decimal
tostring(total)                                   --> "64.92"
decimal.format(total, { symbol = "$" })           --> "$64.92"
```

- `+`, `-`, `*` and unary `-` give a `Decimal`. `+` and `-` use the larger scale of the
  two; `*` adds the scales together (up to 18).
- `==`, `<`, `<=`, `>` and `>=` compare **by value**, so
  `decimal.new("1") == decimal.new("1.000")` is true.
- `..`, `tostring` and `print` show the standard form (`"12.34"`).
- An **integer** works alongside a decimal without any loss (`price * 3`, `total > 0`). A
  **floating-point number** doesn't: `price + 0.1` is a type error, because a binary
  fraction has no exact decimal value. Say what you mean with `decimal.new(0.1, 2)`. This
  is the whole reason the type exists.
- **`/`, `%` and `^` are deliberately not operators.** `1/3` has no exact decimal, so how
  many places to keep is a business decision, not a default: `decimal.div(a, b, 2)`.

```lua
local third = decimal.div(decimal.new("1"), decimal.new("3"), 4)
tostring(third)                                   --> "0.3333"
tostring(decimal.new(0.1, 2) + decimal.new("0.2"))  --> "0.30"
```

<!-- example {"check": "'/' expects numbers, got Decimal"} -->
```lua
local share = decimal.new("1") / 3                -- reported: use decimal.div
```

Sealed means a decimal can only come from `decimal.new` (or from one of the app's
functions that returns one). A table you build yourself, like `{units = 1999, scale = 2}`,
is **not** a decimal: the operators refuse it, and so does every `decimal.*` function.
The one exception is `decimal.new(t)`, which deliberately **accepts** such a table. That
is how a decimal the app passes in as plain data becomes a real decimal again.

This doesn't bring back metatables or redefinable operators: `setmetatable` refuses an
`__add` (or any other operator field) whenever a script tries to set one, and only the
app can create a sealed type.

- `decimal.new(value [, scale])` makes a decimal from a string (`"19.99"`), an integer,
  another decimal, or a floating-point number (which **needs** a scale, because binary
  fractions are inexact). With a `scale`, the value is rounded, halves away from zero.
- `decimal.add` and `sub` (matching scales), `mul` (adding scales, up to 18),
  `div(a, b, scale)` (rounding halves away from zero; dividing by zero is a
  `divisionByZero` error), `neg`, `abs` and `rescale(a, scale)`.
- `decimal.compare` (−1, 0 or 1), `eq`, `lt` and `lte`.
- `decimal.tostring` (the standard form, such as `"12.34"`), `tonumber` (a
  floating-point number, which may lose precision) and `format(a [, opts])` (with digit
  grouping and an optional currency symbol; `opts` can set `scale`, `symbol`,
  `grouping`, `thousands_sep` and `decimal_sep`).

```lua
local big = decimal.new("1234567.891")
decimal.format(big, { scale = 2 })                                 --> "1,234,567.89"
decimal.format(big, { scale = 2, thousands_sep = ".", decimal_sep = "," })  --> "1.234.567,89"
decimal.compare(decimal.new("2"), decimal.new("10"))               --> -1
tostring(decimal.rescale(decimal.new("2.5"), 0))                   --> "3"
tostring(decimal.new({ units = 1999, scale = 2 }))                 --> "19.99"
```

## `stats`

Summary statistics over a list (the table's array part, in order). An element that isn't
a number is a `typeMismatch` error, and an empty list where a value is needed is an
`invalidArgument` error.

- `stats.sum` (an integer when the result is exact), `mean`, `min` and `max` (keeping
  the element's type), `range`, `median` (the middle value for an odd count, the average
  of the two middle values for an even one) and `mode` (the smallest value wins a tie).
- `stats.variance(t [, sample])` and `stddev(t [, sample])`: the population figure by
  default, or the sample figure (dividing by n−1) with `sample = true` (at least two
  values needed).
- `stats.percentile(t, p)`: the p-th percentile (0–100), interpolating linearly between
  values (the method spreadsheets usually use).

```lua
local xs = { 4, 8, 15, 16, 23, 42 }
stats.sum(xs)              --> 108
stats.mean(xs)             --> 18.0
stats.median(xs)           --> 15.5
stats.range(xs)            --> 38
stats.percentile(xs, 50)   --> 15.5
stats.mode({ 3, 1, 3, 1 }) --> 1
stats.mean({})             --> error: empty
```

## `template`

A simple **Mustache** template engine: text with tags, filled in from a table of data.

- `template.render(template, data [, partials])` gives a string.
- Tags: `{{name}}` (with `& < > " '` escaped for HTML), `{{{name}}}` or `{{&name}}`
  (not escaped), `{{#s}}…{{/s}}` (a section), `{{^s}}…{{/s}}` (an inverted section),
  `{{>p}}` (a partial, looked up in the `partials` table of names to templates),
  `{{! comment }}`, dotted paths (`{{a.b.c}}`) and `{{.}}` for the current item.
- A section is skipped for `nil`, `false` or an empty table. A non-empty list repeats the
  section for each item, with the item in scope; any other table renders it once, with
  the table in scope. Names not found in the current scope are looked up in the outer
  ones.
- A missing name or partial renders as nothing. Unbalanced or mismatched tags are an
  `invalidArgument` error, and partials can only nest so deep, so a partial that includes
  itself fails with an error instead of looping.
- **Unlike standard Mustache**, a tag alone on its line doesn't remove that line's
  whitespace: a section or comment on its own line keeps the line break.

```lua
local items = { { name = "widget", qty = 3 }, { name = "gizmo", qty = 10 } }
local list = template.render("{{#items}}{{name}}: {{qty}}; {{/items}}", { items = items })
list  --> "widget: 3; gizmo: 10; "
template.render("Hi {{who}}", { who = "<Ada>" })     --> "Hi &lt;Ada&gt;"
template.render("Hi {{{who}}}", { who = "<Ada>" })   --> "Hi <Ada>"
template.render("{{^items}}none{{/items}}", { items = {} })  --> "none"
template.render("{{>line}}", { n = 2 }, { line = "n={{n}}" })  --> "n=2"
```

## `bit32`

Unsigned 32-bit bitwise operations, as in Lua 5.2. Values are taken modulo 2^32 and every
result is in `0…2^32-1`, so nothing here can overflow or fail. The library adds functions,
not operators: `& | ~ << >>` are still not part of the language.

- `band(a, b [, c [, d]])`, `bor(...)` and `bxor(...)` take up to four values.
- `bnot(x)`.
- `lshift(x, disp)` and `rshift(x, disp)` shift logically. Shifting by 32 or more gives
  `0` (as in Lua, and not an error), and a negative shift goes the other way.
- `arshift(x, disp)` shifts arithmetically: the top bit is copied in, so
  `arshift(0xFFFFFFFF, 31)` is `0xFFFFFFFF`.
- `lrotate(x, disp)` and `rrotate(x, disp)` rotate within 32 bits; any amount, however
  large or negative, is allowed.
- `extract(n, field [, width])` and `replace(n, v, field [, width])` read and write bit
  fields. `field` must be at least 0, `width` at least 1 and `field + width` at most 32;
  anything else is an `invalidArgument` error.

```lua
bit32.band(0xFF, 0x0F)              --> 15
bit32.bor(1, 2, 4)                  --> 7
bit32.lshift(1, 31)                 --> 2147483648
bit32.lshift(1, 32)                 --> 0
bit32.arshift(0xFFFFFFFF, 31)       --> 4294967295
bit32.extract(0xABCD, 4, 8)         --> 188

local function xorshift32(x)
  x = bit32.bxor(x, bit32.lshift(x, 13))
  x = bit32.bxor(x, bit32.rshift(x, 17))
  return bit32.bxor(x, bit32.lshift(x, 5))
end
xorshift32(1)                       --> 270369
```

## `hash`

Checksums and digests of the text's UTF-8 bytes, giving the same result on every
platform.

- `hash.crc32(s)` gives a CRC-32 (IEEE) as an integer, and `hash.fnv1a(s)` a 32-bit
  FNV-1a as an integer: fast, for spreading items across buckets, but not secure.
- `hash.sha1(s)` gives 40 hex characters (only for working with systems that require
  it), `hash.sha256(s)` 64 hex characters, and `hash.hmac_sha256(key, message
  [, key_encoding])` 64 lowercase hex characters.
- `key_encoding` says how to read `key`: `"utf8"` (the default) uses its text as-is,
  `"hex"` and `"base64"` decode it to raw bytes first. That lets one HMAC's result key
  the next, as chained signing schemes such as AWS Signature V4 require. An unknown
  encoding, or a key that doesn't decode, is an error.
- `hash.verify_hmac_sha256(key, message, signature [, key_encoding])` checks a
  signature of 64 hex characters (either case) and gives `true` or `false`. The
  comparison takes the same time however many characters match, so it doesn't leak
  the right answer. A malformed signature gives `false`, not an error; a bad
  `key_encoding` or key is still an error, as for `hmac_sha256`.

```lua
hash.crc32("hello")        --> 907060870
hash.sha256("abc")         --> "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"
#hash.hmac_sha256("key", "message")   --> 64
hash.crc32("A") % 16       --> 11
```

Services and webhooks often send or expect the signature in Base64 rather than hex;
`base64.to_hex` and `hex.to_base64` convert between the two:

```lua
local secret, body = "key", "message"
-- A service expecting a Base64 HMAC:
hex.to_base64(hash.hmac_sha256(secret, body))   --> "bp7ym3X//Ft6uuUn1Y/a2y/kLnIZARl2kXNDBl9Y7Uo="
-- A webhook that sends a Base64 signature:
local signature = "bp7ym3X//Ft6uuUn1Y/a2y/kLnIZARl2kXNDBl9Y7Uo="
hash.verify_hmac_sha256(secret, body, base64.to_hex(signature))   --> true
hash.verify_hmac_sha256(secret, body, "not a signature")          --> false
hash.hmac_sha256(secret, body, "latin1")   --> error: key_encoding
```

Chained signing passes each step's digest on as the next key, with `"hex"` (this is
the AWS Signature V4 signing key from AWS's own documentation):

```lua
local k = hash.hmac_sha256("AWS4wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20150830")
k = hash.hmac_sha256(k, "us-east-1", "hex")
k = hash.hmac_sha256(k, "iam", "hex")
k = hash.hmac_sha256(k, "aws4_request", "hex")
k   --> "c4afb1cc5771d871763a393e44b703571b55cc28424d1a5e86da6ed3c154a4b9"
```

## `tpath`

Reads and writes nested tables using a path written with dots and brackets, such as
`"items[2].sku"`. Positions start at **1**, as elsewhere in Lua. A badly formed path is an
`invalidArgument` error.

- `tpath.get(t, path [, default])` gives the value, or `default` (`nil` unless given) when
  any step is missing.
- `tpath.set(t, path, value)` sets the value and returns `t`, creating tables along the
  way (passing through an existing value that isn't a table is an error).
- `tpath.exists(t, path)` gives a boolean, and `tpath.delete(t, path)` removes the value
  and returns `t`.

```lua
local order = { items = { { sku = "A" }, { sku = "B" } } }
tpath.get(order, "items[2].sku")        --> "B"
tpath.get(order, "items[9].sku", "?")   --> "?"
local cfg = {}
tpath.set(cfg, "retry.maxAttempts", 3)
tpath.get(cfg, "retry.maxAttempts")     --> 3
tpath.exists(cfg, "retry.delay")        --> false
```

## `validate`

Checks a value against a schema written as a table in the style of JSON Schema: useful
for data decoded from JSON or CSV, whose shape isn't known until the script runs.

- `validate.check(value, schema)` gives two values, `(ok, errors)`: a boolean, and a list
  of `{path: string, message: string}` records (empty when `ok` is true). The usual form
  is `local ok, errors = validate.check(v, schema)`.
- `validate.is_valid(value, schema)` gives just the boolean.
- Supported keywords: `type` (a string or a list of `null`, `boolean`, `integer`,
  `number`, `string`, `array` and `object`), `enum`, `const`, `required`, `properties`,
  `additionalProperties` (a boolean or a schema), `items`, `minItems` and `maxItems`,
  `uniqueItems`, `minimum`, `maximum`, `exclusiveMinimum` and `exclusiveMaximum`,
  `minLength` and `maxLength`, `pattern`, and `anyOf`, `allOf`, `oneOf` and `not`.
- **Unlike JSON Schema**, `pattern` is a **Lua pattern**, not a regular expression; a
  non-empty list counts as an `array` (only an empty table, or one with keys that aren't
  positions, is an `object`); keywords it doesn't know are ignored rather than being
  errors; and `enum` and `const` compare simple values. Schemas can nest only so deep
  (going deeper is an error you can catch).

```lua
local schema = {
  type = "object",
  required = { "id" },
  properties = { id = { type = "integer" }, items = { type = "array", minItems = 1 } },
}
validate.is_valid({ id = 7, items = { "A" } }, schema)   --> true
local ok, errors = validate.check({ items = {} }, schema)
ok          --> false
#errors     --> 2
```

## `text`

Reshapes strings. Changing case gives the same result in every language setting, and
word boundaries and slugs deal in ASCII letters, so results are the same on every
platform.

- Case: `text.camel`, `text.pascal`, `text.snake`, `text.kebab`, `text.title` and
  `text.capitalize` (the first character in capitals, the rest unchanged). Words are split
  at separators and at changes of case, including runs of capitals (`HTTPServer` becomes
  `http_server`).
- `text.slug(s)` gives a lowercase ASCII slug: other characters are dropped, runs of
  anything that isn't a letter or digit become a single `-`, and there are no dashes at
  either end.
- `text.truncate(s, len [, ellipsis])` keeps at most `len` characters and adds "…" when it
  shortens; `text.wrap(s, width)` wraps words greedily; and `text.dedent(s)` and
  `text.indent(s, prefix)` remove and add indentation.
- `text.ordinal(n)` gives "1st", "22nd" and so on, and
  `text.pluralize(word, count [, plural])` adds s, es or ies in the usual English way
  (pass `plural` for irregular words).

```lua
text.snake("HTTPServerName")          --> "http_server_name"
text.camel("order line items")        --> "orderLineItems"
text.slug("Hello, World!")            --> "hello-world"
text.truncate("a very long title", 10) --> "a very lon…"
text.ordinal(3)                       --> "3rd"
text.pluralize("box", 2)              --> "boxes"
text.pluralize("person", 2, "people") --> "people"
```