Verified Commit 9926663d authored by Karel Koci's avatar Karel Koci 🤘

Integrate new URI implementation

This commit changes a lot of functionality including user visible ones
such as configuration language.

The most significant from users point is that in configuration language
functions no longer return handlers for configuration they created. This
is to simplify implementation. The correct approach is to use package or
repository name instead of handler. Handler usage was less powerful and
because of sandboxing also unusable across multiple scripts.

There are additional changes in form of obsoleted extra options for
configuration commands. Repository's extra option subdirs is obsoleted
and works differently and much more. Please review changes in language
documentation file.
parent a718f158
......@@ -14,13 +14,13 @@ supported version of Lua on OpenWRT is 5.1, but there should be very
little difference in what we use). Just the set of functions available
is limited to the functions listed here.
Note that using conditions, loops and variables is fully supported
TIP: Using conditions, loops and variables is fully supported
and sometimes desirable.
Security levels
---------------
There are different security levels of the scripts used. The security
There are different security levels of the scripts used. The security
level may further limit the set of commands and the abilities of given
commands. This is to ensure the server may never send malicious
commands covertly (it still can send version of package that contains
......@@ -37,12 +37,10 @@ Full::
(including compiled `.so` modules). This is what the internals of the
updater would be built in.
Local::
It is possible to reference local files and directories as further
configuration scripts. It is possible to read UCI configuration and
execute arbitrary shell commands.
It is possible to read UCI configuration and execute arbitrary
shell commands.
Remote::
The functions may reference only other remote resources, not local
ones. Reading UCI config is not possible.
Reading UCI config is not possible.
Restricted::
It is possible to further restrict the entities referenced to a
string match (eg. ensure that it comes from a given server). Access
......@@ -161,16 +159,14 @@ URIs specified in these verification options are not verified (default values, n
inherited ones, are used). Because of that it is suggested to used only
trusted/secure URIs for that purpose. Suggested are `file://` and `data://`.
NOTE::
Another option `verification` exist. It was originally used for verification
level specification but that is now replaced with `pubkey` and `ca` option
specific values. For backward compatibility it is silently ignored.
NOTE: Another option `verification` exist. It was originally used for verification
level specification but that is now replaced with `pubkey` and `ca` option
specific values. For backward compatibility it is silently ignored.
NOTE::
For `ca` option there is also constant `system_cas` and for `crl` option there
is constant `no_crl`. These are obsoleted but are still defined. `system_cas` is
defined as `true` and `no_crl` is defined as `false`.
`system_cas`
NOTE: For `ca` option there is also constant `system_cas` and for `crl` option
there is constant `no_crl`. These are obsoleted but are still defined.
`system_cas` is defined as `true` and `no_crl` is defined as `false`.
`system_cas`
Dependency description
----------------------
......@@ -213,12 +209,10 @@ Most of the commands has following format:
Command("string", "string", {param = 1})
They start with a capital letter, since they act as constructors.
Script
~~~~~~
script = Script("uri", { extra })
Script("uri", { extra })
This command runs another script.
......@@ -234,14 +228,8 @@ security::
raise the level, such attempt is reported as an error. If not
specified, the level is deduced from the URI. If the URI is remote,
it doesn't go above `remote`, otherwise it doesn't go above `local`.
ignore::
Ignore certain errors. If they happen, don't process such script,
but continue with the rest. This is a lua table with strings, each
one specifying a category of erorrs to ignore.
missing;;
If the script can't be found.
integrity;;
Some signatures don't match.
optional::
Set this to `true` to not fail if retrieval of this script fails.
sig::
pubkey::
ca::
......@@ -249,67 +237,44 @@ crl::
ocsp::
Options to verify the script integrity.
Note that following format is now marked as obsolete and should not be used:
WARNING: Following format is now marked as obsolete and should not be used:
`Script("script-name", "uri", { extra })`
script = Script("script-name", "uri", { extra })
NOTE: There is also obsoleted extra option `ignore`. This should not be used and
any value set to it is effectively considered to be same as setting `optional` to
`true`.
Repository
~~~~~~~~~~
repository = Repository("repository-name", "uri", { extra })
Repository("repository-name", "uri", { extra })
This command introduces another repository of packages. The name may
This command introduces another repository of packages. The can
be used as a reference from other commands and is used in error
messages. However, every place where the name may be used, the result
of the command may be used instead. It is legal to have multiple
repositories with the same name, but referencing it by name may
produce any of them. Referencing by the result is reliable.
messages. Be aware that collision names are considered error and
such repositories are not considered.
The URI is expected to contain an OpenWRT repository in the format
produced by the buildroot.
Extra parameters are:
subdirs::
If the URI contains multiple subdirectories, each one being a valid
repository, you may list the subdirectories here (as a lua table of
strings). The repository will unify all the subdirectory contents
together to form one huge repository. In case of collision of
packages between the subdirectories, the first one containing a
given package wins (in the order listed here). If this option is not
listed, the repository acts in normal way (the URI directly
containing the packages).
index::
Overrides the URI at which the repository index lives and uses the
main URI as the place where packages are downloaded from. Both
gzipped and plain versions may be in the given URI. If the option is
not listed, it is expected to be in `Packages.gz`. Overriding the
URI is not compatible with the subdirs option.
ignore::
Ignore certain errors. This is a lua table with strings, each
specifying a category of errors to ignore. If there's an error
ignored, the repository acts as being empty. Otherwise, such error
would cause the updater to stop.
missing;;
This error happens if the repository is not found. This can mean,
for example, that the `https` URI where the index
(`https://example.org/repository/Packages.gz`) returns 404.
However, a missing package from the repository is not this kind of
error (and cannot be ignored, because it is discovered late after
planning what to install).
integrity;;
This is when the integrity verification/signature check fails.
This may be caused by manipulation with the content, or by missing
a key on our side.
syntax;;
It happens when the repository index can not be parsed because of
syntax errors.
not listed, it is expected to be in `Packages.gz`.
priority::
In case of a package being available in multiple directories, the
package is taken from the repository with highest priority. In case
of equality, the one introduced first wins. The default when the
option is not specified is 50. The number must be an integer between
0 and 100.
optional::
Set this to `true` to not fail if it is not possible to receive repository for
any reason or to parse it. This can be due to missing resource or invalid
verification but in both cases this is not fatal for updater execution and it
continues without this repository.
sig::
pubkey::
ca::
......@@ -317,6 +282,18 @@ crl::
ocsp::
Options to verify the index integrity.
NOTE: There is also obsoleted `subdirs` extra parameter. It was intended to be
used as a simple way to add multiple repositories at once. It had small trick
under its sleeve that it combined all those repositories under one name but that
effectively changes nothing. In new versions of Updater-ng this option is only
emulated and repository with name generated with following script is added
instead: `NAME-SUBDIR` where `NAME` is name of repository and `SUBDIR` is specific
`subdirs` values.
NOTE: There is also obsoleted extra option `ignore`. This should not be used and
any value set to it is effectively considered to be same as setting `optional` to
`true`.
Uninstall
~~~~~~~~~
......@@ -325,8 +302,8 @@ Uninstall
This command takes multiple package names. It ensures none of the
packages is installed.
Note that this is not needed most of the time, since unneeded packages
are removed automatically.
TIP: This is not needed most cases, since unneeded packages are removed
automatically.
Extra options modify the packages preceding them, but only up to the
previous extra options block. Therefore, the first two packages in the
......@@ -382,25 +359,23 @@ critical::
network. Other packages may stop working if the update is
interrupted at the wrong time (for example by a power outage), but
would be fixed by another finished updater run.
ignore::
Ignore certain errors regarding the installation request. Note that
errors related to the package itself are modified by the `Package`
command. This takes an array of strings, each string represents one
category of errors to ignore.
missing;;
Don't fail on the package not being available. The package
wouldn't be installed if not available, but the run of the updater
wouldn't be aborted.
Note that a package may be required to be installed or uninstalled
multiple times (for example by multiple scripts). All such
requirements are tried to be met (eg. by unifying the version options,
etc).
optional::
Set this to `true` to not fail if packages is not available from any configured
repository. Be aware that this has implications if form of possible removed
packages from system.
IMPORTANT: Package may be required to be installed or uninstalled multiple times
(for example by multiple scripts). All such requirements are tried to be met (eg.
by unifying the version options, etc).
NOTE: There is also obsoleted but still working option `ignore` which if set to
any boolean true value it is considered as if `optional` extra option would be
set to `true`.
Package
~~~~~~~
package = Package("name", { extra })
Package("name", { extra })
This command allows amending a package from a repository. It allows
for adding dependencies (even negative or alternative dependencies).
......@@ -411,13 +386,6 @@ doesn't really exist, but can participates in the dependency computation.
A package may be amended multiple times. Each time the options are
merged into the package options.
The result may be used instead of a package name in the dependencies
of other packages and in `Install` and `Uninstall` commands.
Also, the name parameter is optional. If it is omitted (either
specified as nil or just left out), an unique name is generated. This
is useful only for virtual packages.
The options are:
virtual::
......@@ -486,19 +454,10 @@ abi_change_deep::
package that changed its ABI. That means if some package is reinstalled because
of change of ABI, all packages that depends on it are also reinstalled and so
on.
ignore::
Ignore listed categories of errors. This takes an array of strings,
each string meaning one category to ignore.
deps;;
Don't error on missing dependencies. Simply install the package
without satisfying the dependency.
validation;;
Install the package despite it failing validation (eg. when having
different checksum).
installation;;
Don't report errors of installation in this package as an error
and don't abort the rest of the installation process even if it is
in an early stage.
NOTE: Originally there was also option `ignore` that allowed pass for different
problems but most of those were not working and usage of them was questionable.
This options is now considered as obsolete and is ignored.
Export and Unexport
~~~~~~~~~~~~~~~~~~~
......@@ -620,6 +579,9 @@ abi_change_deep::
replan_string::
Updater expects replan to be a string (if this feature isn't set than it's
expected to be only boolean).
no_returns::
Functions such as `Repository` and `Package` no longer return handler that can
be used in other calls.
installed
~~~~~~~~~
......
......@@ -35,10 +35,11 @@ local mkdir = mkdir
local stat = stat
local events_wait = events_wait
local run_util = run_util
local uri = require "uri"
module "utils"
-- luacheck: globals lines2set map set2arr arr2set cleanup_dirs dir_ensure mkdirp read_file write_file clone shallow_copy table_merge arr_append exception multi_index private filter_best strip table_overlay randstr arr_prune arr_inv file_exists
-- luacheck: globals lines2set map set2arr arr2set cleanup_dirs dir_ensure mkdirp read_file write_file clone shallow_copy table_merge arr_append exception multi_index private filter_best strip table_overlay table_wrap randstr arr_prune arr_inv file_exists uri_syste_cas uri_no_crl uri_config uri_content
--[[
Convert provided text into set of lines. Doesn't care about the order.
......@@ -357,6 +358,18 @@ function table_overlay(table)
})
end
--[[
This function returns always table. If input is not table then it is placed to
table. If input is table then it is returned as is.
]]
function table_wrap(table)
if type(table) == "table" then
return table
else
return {table}
end
end
--[[
Check whether file exists
]]
......@@ -370,4 +383,51 @@ function file_exists(name)
end
end
--[[
This function applies given table of configuration to given uri object.
This is here because we need bridge between old approach of using lua tables and
approach of inherited settings in uri object.
For full support of all fields see language documentation, section Verification.
Any field that is not set in table is ignored (configuration is not changed).
]]
function uri_config(uriobj, config)
-- TODO and how about veri?
if config.ca ~= nil then
uriobj:set_ssl_verify(config.ca)
uriobj:add_ca(nil)
for ca in pairs(table_wrap(config.ca)) do
uriobj:add_ca(ca)
end
end
if config.crl ~= nil then
uriobj:add_crl(nil)
for crl in pairs(table_wrap(config.crl)) do
uriobj:add_crl(crl)
end
end
if config.ocsp ~= nil then
uriobj:set_ocsp(config.ocsp)
end
if config.pubkey ~= nil then
uriobj:add_pubkey(nil)
for pubkey in pairs(table_wrap(config.pubkey)) do
uriobj:add_pubkey(pubkey)
end
end
if config.sig ~= nil then
uriobj:set_sig(config.sig)
end
end
-- Get content of given URI
-- It returns downloaded content as first argument and uri object as second (which
-- can be used as a parent to other uris)
function uri_content(struri, parent, config)
local master = uri.new()
local u = master:to_buffer(struri, parent)
uri_config(u, config)
-- TODO finish error and others?
return u:finish(), u
end
return _M
......@@ -24,21 +24,25 @@ the configuration scripts to be run in.
local pairs = pairs
local ipairs = ipairs
local next = next
local type = type
local pcall = pcall
local string = string
local error = error
local require = require
local tostring = tostring
local assert = assert
local table = table
local unpack = unpack
local utils = require "utils"
local uri = require "uri"
local DBG = DBG
local WARN = WARN
local ERROR = ERROR
module "requests"
-- luacheck: globals known_packages package_wrap known_repositories known_repositories_all repo_serial repository repository_get content_requests install uninstall script package
-- luacheck: globals known_packages known_repositories repositories_uri_master repo_serial repository content_requests install uninstall script package
-- Verifications fields are same for script, repository and package. Lets define them here once and then just append.
local allowed_extras_verification = {
......@@ -95,19 +99,6 @@ local function extra_check_package_type(pkg, field)
end
end
-- Common check for accepted values in table
local function extra_check_table(field, what, table, accepted)
local acc = utils.arr2set(accepted)
for _, v in pairs(table) do
if type(v) ~= "string" then
extra_field_invalid_type(v, field, what)
end
if not acc[v] then
WARN("Unknown value " .. v .. " in table of extra option " .. field .. " for a " .. what)
end
end
end
-- Common check for verification field
local function extra_check_verification(what, extra)
if extra.verification == nil then return end -- we don't care if there is no setting
......@@ -140,7 +131,7 @@ local allowed_package_extras = {
["abi_change"] = utils.arr2set({"table", "boolean"}),
["abi_change_deep"] = utils.arr2set({"table", "boolean"}),
["priority"] = utils.arr2set({"number"}),
["ignore"] = utils.arr2set({"table"})
["ignore"] = utils.arr2set({"table"}), -- obsoleted
}
utils.table_merge(allowed_package_extras, allowed_extras_verification)
......@@ -165,8 +156,7 @@ local function extra_check_deps(what, field, deps)
end
else
invalid(deps)
end
end
end end
--[[
We simply store all package promises, so they can be taken
......@@ -176,6 +166,16 @@ We just store them in an array for future processing.
]]
known_packages = {}
local function new_package(pkg_name, extra)
local pkg = {
tp = "package",
name = pkg_name,
}
utils.table_merge(pkg, extra)
table.insert(known_packages, pkg)
return pkg
end
--[[
This package is just a promise of a real package in the future. It holds the
name and possibly some additional info for the package. Once we go through
......@@ -187,7 +187,7 @@ has been run).
The package has no methods, it's just a stupid structure.
]]
function package(_, pkg, extra)
function package(_, pkg_name, extra)
-- Minimal typo verification. Further verification is done when actually using the package.
extra = allowed_extras_check_type(allowed_package_extras, "package", extra or {})
extra_check_verification("package", extra)
......@@ -218,53 +218,34 @@ function package(_, pkg, extra)
extra_check_package_type(v, name)
end
end
elseif name == "ignore" then
extra_check_table("package", name, value, {"deps", "validation", "installation"})
end
end
local result = {}
utils.table_merge(result, extra)
result.name = pkg
result.tp = "package"
table.insert(known_packages, result)
return result
end
--[[
Either create a new package of that name (if string is passed) or
pass the provided package.
]]
function package_wrap(context, pkg)
if type(pkg) == "table" and pkg.tp == "package" then
-- It is already a package object
return pkg
else
return package(context, pkg)
if extra["ignore"] then -- obsolete
WARN('Package extra option "ignore" is obsolete and is ignored.')
extra["ignore"] = nil
end
new_package(pkg_name, extra)
end
-- List of allowed extra options for a Repository command
local allowed_repository_extras = {
["subdirs"] = utils.arr2set({"table"}),
["index"] = utils.arr2set({"string"}),
["ignore"] = utils.arr2set({"table"}),
["priority"] = utils.arr2set({"number"}),
["optional"] = utils.arr2set({"boolean"}),
["subdirs"] = utils.arr2set({"table"}), -- obsolete
["ignore"] = utils.arr2set({"table"}), -- obsolete
}
utils.table_merge(allowed_repository_extras, allowed_extras_verification)
--[[
The repositories we already created. If there are multiple repos of the
same name, we are allowed to provide any of them. Therefore, this is
indexed by their names.
]]
-- All added known repositories
known_repositories = {}
-- One with all the repositories, even if there are name collisions
known_repositories_all = {}
-- Order of the repositories as they are parsed
-- Order of the repositories as they are introduced
-- We need this to decide in corner case of same repository priority
repo_serial = 1
repositories_uri_master = uri.new()
--[[
Promise of a future repository. The repository shall be downloaded after
all the configuration scripts are run, parsed and used as a source of
......@@ -275,99 +256,78 @@ function repository(context, name, repo_uri, extra)
-- Catch possible typos
extra = allowed_extras_check_type(allowed_repository_extras, 'repository', extra or {})
extra_check_verification("repository", extra)
for name, value in pairs(extra) do
if name == "subdirs" or name == "ignore" then
for _, v in pairs(value) do
if type(v) ~= "string" then
extra_field_invalid_type(v, name, "repository")
end
end
elseif name == "ignore" then
extra_check_table("repository", name, value, {"missing", "integrity", "syntax"})
if extra.ignore then
WARN('Repository extra option "ignore" is obsolete and should not be used. Use "optional" instead.')
if extra.optional == nil then
extra.optional = next(extra.ignore) ~= nil -- if any ignore was specified then set it as optional
end
extra.ignore = nil
end
local result = {}
utils.table_merge(result, extra)
result.repo_uri = repo_uri
utils.private(result).context = context
--[[
Start the download. This way any potential access violation is reported
right away. It also allows for some parallel downloading while we process
the configs.
Pass result as the validation parameter, as all validation info would be
part of the extra.
We do some mangling with the sig URI, since they are not at Package.gz.sig, but at
Package.sig only.
]]
if extra.subdirs then
utils.private(result).index_uri = {}
for _, sub in pairs(extra.subdirs) do
sub = "/" .. sub
local u = repo_uri .. sub .. '/Packages.gz'
local params = utils.table_overlay(result)
params.sig = repo_uri .. sub .. '/Packages.sig'
utils.private(result).index_uri[sub] = uri(context, u, params)
local function register_repo(u, repo_name)
if known_repositories[repo_name] then
ERROR("Repository of name '" .. repo_name "' was already added. Repetition is ignored.")
return
end
else
local u = result.index or repo_uri .. '/Packages.gz'
local params = utils.table_overlay(result)
params.sig = params.sig or u:gsub('%.gz$', '') .. '.sig'
utils.private(result).index_uri = {[""] = uri(context, u, params)}
local iuri = repositories_uri_master:to_buffer(u, context.paret_script_uri)
utils.uri_config(iuri, {unpack(extra), ["sig"] = extra.sig or u:gsub('%.gz$', '') .. '.sig'})
local repo = {
tp = "repository",
index_uri = iuri,
repo_uri = repo_uri,
name = repo_name,
serial = repo_serial,
}
utils.table_merge(repo, extra)
repo.priority = extra.priority or 50
known_repositories[repo_name] = repo
repo_serial = repo_serial + 1
end
result.priority = result.priority or 50
result.serial = repo_serial
repo_serial = repo_serial + 1
result.name = name
result.tp = "repository"
known_repositories[name] = result
table.insert(known_repositories_all, result)
return result
end
-- Either return the repo, if it is one already, or look it up. Nil if it doesn't exist.
function repository_get(repo)
if type(repo) == "table" and (repo.tp == "repository" or repo.tp == "parsed-repository") then
return repo
if extra.subdirs then
WARN('Repository extra option "subdirs" is obsolete and should not be used anymore.')
for _, sub in pairs(extra.subdirs) do
register_repo(repo_uri .. '/' .. sub .. '/' .. (extra.index or 'Packages.gz'), name .. '-' .. sub)
end
else
return known_repositories[repo]
register_repo(repo_uri .. '/' .. (extra.index or 'Packages.gz'), name)
end
end
local allowed_install_extras = {
["priority"] = utils.arr2set({"number"}),
["version"] = utils.arr2set({"string"}),
["repository"] = utils.arr2set({"string", "table"}),
["reinstall"] = utils.arr2set({"boolean"}),
["critical"] = utils.arr2set({"boolean"}),
["ignore"] = utils.arr2set({"table"})
}
-- This is list of all requests to be fulfilled
content_requests = {}
local function content_request(context, cmd, allowed, ...)
local function content_request(cmd, allowed, ...)
local batch = {}
local function submit(extras)
for _, pkg in ipairs(batch) do
pkg = package_wrap(context, pkg)
DBG("Request " .. cmd .. " of " .. (pkg.name or pkg))
local request = {
package = pkg,
tp = cmd
}
extras = allowed_extras_check_type(allowed, cmd, extras)
for name, value in pairs(extras) do
if name == "repository" and type(value) == "table" then
for _, v in pairs(value) do
if type(v) ~= "string" then
extra_field_invalid_type(v, name, cmd)
end
extras = allowed_extras_check_type(allowed, cmd, extras)
if extras.repository then
if type(extras.repository) == "table" then
for _, v in pairs(extras.repository) do
if type(v) ~= "string" then
extra_field_invalid_type(v, "repository", cmd)
end
elseif name == "ignore" then -- note: we don't check what cmd we have as allowed_extras_check_type filters out ignore parameters for uninstall
extra_check_table("cmd", name, value, {"missing"})
end
end
end
if extras.ignore then
-- Note: this is applicable only to Install
WARN('Install extra option "ignore" is obsolete and should not be used. Use "optional" instead.')
if extras.optional == nil then
extras.optional = next(extras.ignore) ~= nil -- if any ignore was specified then set it as optional
end
extras.ignore = nil
end
for _, pkg_name in ipairs(batch) do
DBG("Request " .. cmd .. " of " .. pkg_name)
local request = {
package = new_package(pkg_name, {}),
tp = cmd
}
utils.table_merge(request, extras)
request.priority = request.priority or 50
table.insert(content_requests, request)
......@@ -375,7 +335,7 @@ local function content_request(context, cmd, allowed, ...)
batch = {}
end
for _, val in ipairs({...}) do
if type(val) == "table" and val.tp ~= "package" then
if type(val) == "table" then
submit(val)
else
table.insert(batch, val)
......@@ -384,35 +344,36 @@ local function content_request(context, cmd, allowed, ...)
submit({})
end
function install(context, ...)
return content_request(context, "install", allowed_install_extras, ...)
local allowed_install_extras = {
["priority"] = utils.arr2set({"number"}),
["version"] = utils.arr2set({"string"}),
["repository"] = utils.arr2set({"string", "table"}),
["reinstall"] = utils.arr2set({"boolean"}),
["critical"] = utils.arr2set({"boolean"}),
["optional"] = utils.arr2set({"boolean"}),
["ignore"] = utils.arr2set({"table"}), -- obsolete
}
function install(_, ...)
return content_request("install", allowed_install_extras, ...)
end
local allowed_uninstall_extras = {
["priority"] = utils.arr2set({"number"})
}
function uninstall(context, ...)
return content_request(context, "uninstall", allowed_uninstall_extras, ...)
function uninstall(_, ...)
return content_request("uninstall", allowed_uninstall_extras, ...)
end
local allowed_script_extras = {
["security"] = utils.arr2set({"string"}),
["restrict"] = utils.arr2set({"string"}), -- This is now obsoleted (not used)
["ignore"] = utils.arr2set({"table"})
["optional"] = utils.arr2set({"boolean"}),
["restrict"] = utils.arr2set({"string"}), -- obsolete
["ignore"] = utils.arr2set({"table"}), -- obsolete
}
utils.table_merge(allowed_script_extras, allowed_extras_verification)
--[[
We want to insert these options into the new context, if they exist.
]]
local script_insert_options = {
pubkey = true,
ca = true,
crl = true,
ocsp = true
}
--[[
Note that we have filler field just for backward compatibility so when we have
just one argument or two arguments where second one is table we move all arguments
......@@ -427,40 +388,33 @@ function script(context, filler, script_uri, extra)
else
WARN("Syntax \"Script('script-name', 'uri', { extra })\" is deprecated and will be removed.")
end
DBG("Running script " .. script_uri)
extra = allowed_extras_check_type(allowed_script_extras, 'script', extra or {})
extra_check_verification("script", extra)
for name, value in pairs(extra) do
if name == "ignore" then
extra_check_table("script", script_uri, value, {"missing", "integrity"})
if extra.ignore then
WARN('Script extra option "ignore" is obsolete and should not be used. Use "optional" instead.')