scripting / scripting-5Markdown ↓

Extra libraries

Beyond the standard library, an app can offer any of these extra libraries: json, base64, hex, url, csv, decimal, stats, template, bit32, hash, tpath, validate and text. Which ones your scripts can use is up to the app. They follow the same rules as the standard library: every function gives the same result for the same input, none of them reaches files, the network, the clock or randomness, and none can crash the app. Bad input is an ordinary error you can catch (invalidArgument, typeMismatch, divisionByZero and so on); running out of a limit (string or table size, time) can't be caught.

All of them treat strings as text (Unicode characters). Functions that deal in bytes work on the text's UTF-8 bytes, and anything decoded that isn't valid UTF-8 text is refused rather than producing a broken string.

Every function is fully typed, so the checker knows its parameters and results. Where a result depends on an argument's type, the checker connects them: table.clone(t), deepcopy(t) and extend(dst, …), assert(v), typecheck(v, …), and tpath.set(t, …) and delete(t, …) return the same type as their argument.

In the examples, --> shows what a line produces.

json

Turns script values into JSON text and back, with the same result every time.

  • json.encode(value [, opts]) gives a string.
    • opts.indent (1–10) pretty-prints with that many spaces; leaving it out (or 0) gives compact output.
    • opts.sort_keys (a boolean) sorts object keys. Otherwise keys stay in the order they were added, matching pairs.
    • A table becomes a JSON array when its entries are exactly 1…#t, and an object otherwise (number keys become strings). An empty table becomes [].
    • Functions, infinite or NaN numbers, object keys that aren't strings or numbers, and tables that contain themselves are invalidArgument errors, as is nesting deeper than 200 levels.
  • json.decode(s) gives a value. Objects become tables with string keys, arrays become lists starting at 1, and null becomes nil, so a null field is simply absent. Numbers keep their form (42 is an integer; 42.0 and 1e3 are floating-point). \uXXXX escapes, including surrogate pairs, are supported. Badly formed input, or nesting deeper than 200 levels, is an invalidArgument error giving the position.
local order = json.decode('{"id": 7, "items": [{"sku": "A", "qty": 2}], "note": null}')
order.items[1].sku                          --> "A"
order.note                                  --> nil
json.encode({ b = 1, a = { 1, 2 } })        --> '{"b":1,"a":[1,2]}'
json.encode({ b = 1, a = 2 }, { sort_keys = true })  --> '{"a":2,"b":1}'
json.encode({})                             --> "[]"
json.decode("{oops")                        --> error: at position 2

Keeping JSON exactly. Standard Lua has no null, so by default a null field simply disappears, and an empty table encodes as []. When the difference matters, for example when the JSON is checked against a schema, use the explicit forms:

  • json.decode(s, { explicit_null = true }) keeps the document exactly: null becomes json.null(), and every array and object remembers which it was, so encoding the result gives the same JSON back.
  • json.null() is JSON's null as a value a table can hold, and json.isnull(v) tests for it. Compare with json.isnull, not ==.
  • json.array([t]) and json.object([t]) make a table that always encodes as an array or an object, even when it's empty.

These are sealed: a table you build yourself can't pass for one.

local doc = json.decode('{"note": null, "tags": [], "meta": {}}', { explicit_null = true })
json.isnull(doc.note)                       --> true
json.encode(doc, { sort_keys = true })      --> '{"meta":{},"note":null,"tags":[]}'
json.decode('{"note": null}').note          --> nil
json.encode({ items = json.array(), extra = json.object() }, { sort_keys = true })  --> '{"extra":{},"items":[]}'
json.encode({ value = json.null() })        --> '{"value":null}'
json.encode(json.array({ 1, 2 }))           --> "[1,2]"
json.isnull({})                             --> false

base64

  • base64.encode(s [, urlsafe]) gives Base64 text (RFC 4648). With urlsafe = true it uses the -_ alphabet (still padded with =).
  • base64.decode(s [, urlsafe]) gives the decoded text. It accepts either alphabet, and refuses a wrong length, stray characters, or a result that isn't valid UTF-8 text.
  • base64.to_hex(s) gives the lowercase hex of the bytes the Base64 text encodes. It accepts either alphabet and works for any bytes, even ones that aren't UTF-8 text (such as a signature), because they never pass through a Lua string. It refuses a wrong length or stray characters.
base64.encode("user:secret")               --> "dXNlcjpzZWNyZXQ="
base64.decode("dXNlcjpzZWNyZXQ=")          --> "user:secret"
base64.encode("??>", true)                 --> "Pz8-"
base64.to_hex("/w==")                      --> "ff"   -- base64.decode refuses these bytes

hex

  • hex.encode(s) gives lowercase hexadecimal, two digits per byte.
  • hex.decode(s) gives the text back, and refuses an odd length, characters that aren't hex digits, or a result that isn't valid UTF-8 text.
  • hex.to_base64(s [, urlsafe]) gives the Base64 text of the bytes the hex encodes, with urlsafe as in base64.encode. Like base64.to_hex it works for any bytes, so it turns a hex digest into Base64; it refuses an odd length or non-hex characters.
hex.encode("Hi")     --> "4869"
hex.encode("é")      --> "c3a9"
hex.decode("4869")   --> "Hi"
hex.to_base64("ff")  --> "/w=="   -- hex.decode refuses these bytes

url

  • url.encode(s) and url.decode(s) percent-encode a single part of a URL (RFC 3986): letters, digits and -._~ pass through, and everything else becomes %XX for each UTF-8 byte. decode leaves + as it is.
  • url.encode_query(params) gives k=v&… with keys sorted, so the result is always the same, and keys and values percent-encoded. Values can be strings, numbers or booleans.
  • url.decode_query(s) gives a table of the decoded pairs. + becomes a space (as web forms send it), a key with no value maps to "", and a later repeat of a key wins. Badly formed escapes are refused.
url.encode("a b&c")                               --> "a%20b%26c"
url.encode_query({ q = "two words", page = 2 })   --> "page=2&q=two%20words"
url.decode_query("a=1&b=two+words&flag")          --> {a = "1", b = "two words", flag = ""}

csv

Follows RFC 4180. opts.delimiter is a single character ("," unless you say otherwise).

  • csv.parse(s [, opts]) gives a list of rows, each a list of text fields. Quoted fields, delimiters and line breaks inside quotes, and "" for a quote are all handled. Windows and Unix line endings both end a row, and a final line break doesn't add an empty row. Badly formed quoting is read as sensibly as possible rather than failing.
  • csv.format(rows [, opts]) gives CSV text. A field containing the delimiter, a quote or a line break is quoted, with "" for each quote, and rows are joined with "\n". Fields can be strings, numbers or booleans (nil gives an empty field).
local rows = csv.parse("name,qty\nwidget,3\n\"gizmo, large\",10\n")
#rows          --> 3
rows[3][1]     --> "gizmo, large"
csv.format({ { "name", "qty" }, { "re,sold", 5 } })   --> 'name,qty\n"re,sold",5'
csv.parse("a;b", { delimiter = ";" })                 --> {{"a", "b"}}

decimal

Exact decimals, for money and similar values. A decimal can't be changed once made, and has a value and a scale (the number of decimal places, 0–18): 12.34 has scale 2. Every operation checks for overflow and division by zero. The library provides the Decimal type, so the checker follows decimals through your code (decimal.add(a, b) is a Decimal) and refuses tables that aren't decimals.

Decimal is a sealed type (see the Language Guide, §5), so ordinary operators work:

local price, qty, shipping = decimal.new("19.99"), 3, decimal.new("4.95")
local total = price * qty + shipping              -- a Decimal
tostring(total)                                   --> "64.92"
decimal.format(total, { symbol = "$" })           --> "$64.92"
  • +, -, * and unary - give a Decimal. + and - use the larger scale of the two; * adds the scales together (up to 18).
  • ==, <, <=, > and >= compare by value, so decimal.new("1") == decimal.new("1.000") is true.
  • .., tostring and print show the standard form ("12.34").
  • An integer works alongside a decimal without any loss (price * 3, total > 0). A floating-point number doesn't: price + 0.1 is a type error, because a binary fraction has no exact decimal value. Say what you mean with decimal.new(0.1, 2). This is the whole reason the type exists.
  • /, % and ^ are deliberately not operators. 1/3 has no exact decimal, so how many places to keep is a business decision, not a default: decimal.div(a, b, 2).
local third = decimal.div(decimal.new("1"), decimal.new("3"), 4)
tostring(third)                                   --> "0.3333"
tostring(decimal.new(0.1, 2) + decimal.new("0.2"))  --> "0.30"
local share = decimal.new("1") / 3                -- reported: use decimal.div

Sealed means a decimal can only come from decimal.new (or from one of the app's functions that returns one). A table you build yourself, like {units = 1999, scale = 2}, is not a decimal: the operators refuse it, and so does every decimal.* function. The one exception is decimal.new(t), which deliberately accepts such a table. That is how a decimal the app passes in as plain data becomes a real decimal again.

This doesn't bring back metatables or redefinable operators: setmetatable refuses an __add (or any other operator field) whenever a script tries to set one, and only the app can create a sealed type.

  • decimal.new(value [, scale]) makes a decimal from a string ("19.99"), an integer, another decimal, or a floating-point number (which needs a scale, because binary fractions are inexact). With a scale, the value is rounded, halves away from zero.
  • decimal.add and sub (matching scales), mul (adding scales, up to 18), div(a, b, scale) (rounding halves away from zero; dividing by zero is a divisionByZero error), neg, abs and rescale(a, scale).
  • decimal.compare (−1, 0 or 1), eq, lt and lte.
  • decimal.tostring (the standard form, such as "12.34"), tonumber (a floating-point number, which may lose precision) and format(a [, opts]) (with digit grouping and an optional currency symbol; opts can set scale, symbol, grouping, thousands_sep and decimal_sep).
local big = decimal.new("1234567.891")
decimal.format(big, { scale = 2 })                                 --> "1,234,567.89"
decimal.format(big, { scale = 2, thousands_sep = ".", decimal_sep = "," })  --> "1.234.567,89"
decimal.compare(decimal.new("2"), decimal.new("10"))               --> -1
tostring(decimal.rescale(decimal.new("2.5"), 0))                   --> "3"
tostring(decimal.new({ units = 1999, scale = 2 }))                 --> "19.99"

stats

Summary statistics over a list (the table's array part, in order). An element that isn't a number is a typeMismatch error, and an empty list where a value is needed is an invalidArgument error.

  • stats.sum (an integer when the result is exact), mean, min and max (keeping the element's type), range, median (the middle value for an odd count, the average of the two middle values for an even one) and mode (the smallest value wins a tie).
  • stats.variance(t [, sample]) and stddev(t [, sample]): the population figure by default, or the sample figure (dividing by n−1) with sample = true (at least two values needed).
  • stats.percentile(t, p): the p-th percentile (0–100), interpolating linearly between values (the method spreadsheets usually use).
local xs = { 4, 8, 15, 16, 23, 42 }
stats.sum(xs)              --> 108
stats.mean(xs)             --> 18.0
stats.median(xs)           --> 15.5
stats.range(xs)            --> 38
stats.percentile(xs, 50)   --> 15.5
stats.mode({ 3, 1, 3, 1 }) --> 1
stats.mean({})             --> error: empty

template

A simple Mustache template engine: text with tags, filled in from a table of data.

  • template.render(template, data [, partials]) gives a string.
  • Tags: {{name}} (with & < > " ' escaped for HTML), {{{name}}} or {{&name}} (not escaped), {{#s}}…{{/s}} (a section), {{^s}}…{{/s}} (an inverted section), {{>p}} (a partial, looked up in the partials table of names to templates), {{! comment }}, dotted paths ({{a.b.c}}) and {{.}} for the current item.
  • A section is skipped for nil, false or an empty table. A non-empty list repeats the section for each item, with the item in scope; any other table renders it once, with the table in scope. Names not found in the current scope are looked up in the outer ones.
  • A missing name or partial renders as nothing. Unbalanced or mismatched tags are an invalidArgument error, and partials can only nest so deep, so a partial that includes itself fails with an error instead of looping.
  • Unlike standard Mustache, a tag alone on its line doesn't remove that line's whitespace: a section or comment on its own line keeps the line break.
local items = { { name = "widget", qty = 3 }, { name = "gizmo", qty = 10 } }
local list = template.render("{{#items}}{{name}}: {{qty}}; {{/items}}", { items = items })
list  --> "widget: 3; gizmo: 10; "
template.render("Hi {{who}}", { who = "<Ada>" })     --> "Hi &lt;Ada&gt;"
template.render("Hi {{{who}}}", { who = "<Ada>" })   --> "Hi <Ada>"
template.render("{{^items}}none{{/items}}", { items = {} })  --> "none"
template.render("{{>line}}", { n = 2 }, { line = "n={{n}}" })  --> "n=2"

bit32

Unsigned 32-bit bitwise operations, as in Lua 5.2. Values are taken modulo 2^32 and every result is in 0…2^32-1, so nothing here can overflow or fail. The library adds functions, not operators: & | ~ << >> are still not part of the language.

  • band(a, b [, c [, d]]), bor(...) and bxor(...) take up to four values.
  • bnot(x).
  • lshift(x, disp) and rshift(x, disp) shift logically. Shifting by 32 or more gives 0 (as in Lua, and not an error), and a negative shift goes the other way.
  • arshift(x, disp) shifts arithmetically: the top bit is copied in, so arshift(0xFFFFFFFF, 31) is 0xFFFFFFFF.
  • lrotate(x, disp) and rrotate(x, disp) rotate within 32 bits; any amount, however large or negative, is allowed.
  • extract(n, field [, width]) and replace(n, v, field [, width]) read and write bit fields. field must be at least 0, width at least 1 and field + width at most 32; anything else is an invalidArgument error.
bit32.band(0xFF, 0x0F)              --> 15
bit32.bor(1, 2, 4)                  --> 7
bit32.lshift(1, 31)                 --> 2147483648
bit32.lshift(1, 32)                 --> 0
bit32.arshift(0xFFFFFFFF, 31)       --> 4294967295
bit32.extract(0xABCD, 4, 8)         --> 188

local function xorshift32(x)
  x = bit32.bxor(x, bit32.lshift(x, 13))
  x = bit32.bxor(x, bit32.rshift(x, 17))
  return bit32.bxor(x, bit32.lshift(x, 5))
end
xorshift32(1)                       --> 270369

hash

Checksums and digests of the text's UTF-8 bytes, giving the same result on every platform.

  • hash.crc32(s) gives a CRC-32 (IEEE) as an integer, and hash.fnv1a(s) a 32-bit FNV-1a as an integer: fast, for spreading items across buckets, but not secure.
  • hash.sha1(s) gives 40 hex characters (only for working with systems that require it), hash.sha256(s) 64 hex characters, and hash.hmac_sha256(key, message [, key_encoding]) 64 lowercase hex characters.
  • key_encoding says how to read key: "utf8" (the default) uses its text as-is, "hex" and "base64" decode it to raw bytes first. That lets one HMAC's result key the next, as chained signing schemes such as AWS Signature V4 require. An unknown encoding, or a key that doesn't decode, is an error.
  • hash.verify_hmac_sha256(key, message, signature [, key_encoding]) checks a signature of 64 hex characters (either case) and gives true or false. The comparison takes the same time however many characters match, so it doesn't leak the right answer. A malformed signature gives false, not an error; a bad key_encoding or key is still an error, as for hmac_sha256.
hash.crc32("hello")        --> 907060870
hash.sha256("abc")         --> "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"
#hash.hmac_sha256("key", "message")   --> 64
hash.crc32("A") % 16       --> 11

Services and webhooks often send or expect the signature in Base64 rather than hex; base64.to_hex and hex.to_base64 convert between the two:

local secret, body = "key", "message"
-- A service expecting a Base64 HMAC:
hex.to_base64(hash.hmac_sha256(secret, body))   --> "bp7ym3X//Ft6uuUn1Y/a2y/kLnIZARl2kXNDBl9Y7Uo="
-- A webhook that sends a Base64 signature:
local signature = "bp7ym3X//Ft6uuUn1Y/a2y/kLnIZARl2kXNDBl9Y7Uo="
hash.verify_hmac_sha256(secret, body, base64.to_hex(signature))   --> true
hash.verify_hmac_sha256(secret, body, "not a signature")          --> false
hash.hmac_sha256(secret, body, "latin1")   --> error: key_encoding

Chained signing passes each step's digest on as the next key, with "hex" (this is the AWS Signature V4 signing key from AWS's own documentation):

local k = hash.hmac_sha256("AWS4wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20150830")
k = hash.hmac_sha256(k, "us-east-1", "hex")
k = hash.hmac_sha256(k, "iam", "hex")
k = hash.hmac_sha256(k, "aws4_request", "hex")
k   --> "c4afb1cc5771d871763a393e44b703571b55cc28424d1a5e86da6ed3c154a4b9"

tpath

Reads and writes nested tables using a path written with dots and brackets, such as "items[2].sku". Positions start at 1, as elsewhere in Lua. A badly formed path is an invalidArgument error.

  • tpath.get(t, path [, default]) gives the value, or default (nil unless given) when any step is missing.
  • tpath.set(t, path, value) sets the value and returns t, creating tables along the way (passing through an existing value that isn't a table is an error).
  • tpath.exists(t, path) gives a boolean, and tpath.delete(t, path) removes the value and returns t.
local order = { items = { { sku = "A" }, { sku = "B" } } }
tpath.get(order, "items[2].sku")        --> "B"
tpath.get(order, "items[9].sku", "?")   --> "?"
local cfg = {}
tpath.set(cfg, "retry.maxAttempts", 3)
tpath.get(cfg, "retry.maxAttempts")     --> 3
tpath.exists(cfg, "retry.delay")        --> false

validate

Checks a value against a schema written as a table in the style of JSON Schema: useful for data decoded from JSON or CSV, whose shape isn't known until the script runs.

  • validate.check(value, schema) gives two values, (ok, errors): a boolean, and a list of {path: string, message: string} records (empty when ok is true). The usual form is local ok, errors = validate.check(v, schema).
  • validate.is_valid(value, schema) gives just the boolean.
  • Supported keywords: type (a string or a list of null, boolean, integer, number, string, array and object), enum, const, required, properties, additionalProperties (a boolean or a schema), items, minItems and maxItems, uniqueItems, minimum, maximum, exclusiveMinimum and exclusiveMaximum, minLength and maxLength, pattern, and anyOf, allOf, oneOf and not.
  • Unlike JSON Schema, pattern is a Lua pattern, not a regular expression; a non-empty list counts as an array (only an empty table, or one with keys that aren't positions, is an object); keywords it doesn't know are ignored rather than being errors; and enum and const compare simple values. Schemas can nest only so deep (going deeper is an error you can catch).
local schema = {
  type = "object",
  required = { "id" },
  properties = { id = { type = "integer" }, items = { type = "array", minItems = 1 } },
}
validate.is_valid({ id = 7, items = { "A" } }, schema)   --> true
local ok, errors = validate.check({ items = {} }, schema)
ok          --> false
#errors     --> 2

text

Reshapes strings. Changing case gives the same result in every language setting, and word boundaries and slugs deal in ASCII letters, so results are the same on every platform.

  • Case: text.camel, text.pascal, text.snake, text.kebab, text.title and text.capitalize (the first character in capitals, the rest unchanged). Words are split at separators and at changes of case, including runs of capitals (HTTPServer becomes http_server).
  • text.slug(s) gives a lowercase ASCII slug: other characters are dropped, runs of anything that isn't a letter or digit become a single -, and there are no dashes at either end.
  • text.truncate(s, len [, ellipsis]) keeps at most len characters and adds "…" when it shortens; text.wrap(s, width) wraps words greedily; and text.dedent(s) and text.indent(s, prefix) remove and add indentation.
  • text.ordinal(n) gives "1st", "22nd" and so on, and text.pluralize(word, count [, plural]) adds s, es or ies in the usual English way (pass plural for irregular words).
text.snake("HTTPServerName")          --> "http_server_name"
text.camel("order line items")        --> "orderLineItems"
text.slug("Hello, World!")            --> "hello-world"
text.truncate("a very long title", 10) --> "a very lon…"
text.ordinal(3)                       --> "3rd"
text.pluralize("box", 2)              --> "boxes"
text.pluralize("person", 2, "people") --> "people"

Published platform manual · revision dd82be770e47