language.txt 25.2 KB
Newer Older
1 2 3 4
The updater language
====================

This document is about the language describing specific update
5 6 7 8 9 10
scenarios. It is the core configuration language of the updater
itself, listing which packages should be installed, where they come
from and any special quirks needed for specific situations. Other ways
of configuring are possible (like command line requests to install
some packages, or listing things in UCI file), but these are usually
implemented in this language.
11 12 13 14 15 16

The language is, strictly speaking, ordinary Lua (currently the
supported version of Lua on OpenWRT is 5.1, but there should be very
little difference in what we use). Just the set of functions available
is limited to the functions listed here.

17
TIP: Using conditions, loops and variables is fully supported
18
and sometimes desirable.
19

20 21 22
Security levels
---------------

23
There are different security levels of the scripts used. The security
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
level may further limit the set of commands and the abilities of given
commands. This is to ensure the server may never send malicious
commands covertly (it still can send version of package that contains
the malicious code, but that's impossible to prevent with
an auto-updater, but it would at least have to notify about the
package being installed).

Security levels are:

Full::
  The Lua code is not run in any sandbox at all. All functions here
  work without any limits. Also, all Lua libraries are available
  without any limitation and further Lua code can be required
  (including compiled `.so` modules). This is what the internals of the
  updater would be built in.
Local::
40 41
  It is possible to read UCI configuration and execute arbitrary
  shell commands.
42
Remote::
43
  Reading UCI config is not possible.
44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70
Restricted::
  It is possible to further restrict the entities referenced to a
  string match (eg. ensure that it comes from a given server). Access
  to flag storage is restricted only to flags belonging to the current
  script and scripts it references.

No function allows raising the security level when referencing another
script.

Each script runs with its own environment ‒ they don't see each
other's variables.

Order of execution
------------------

The scripts are executed in the order they are referenced, in DFS
order. A referenced script is first fully executed (with its
sub-scripts) before the current script continues. In that sense, it
works similar to any other scripting language `include` command.

However, the execution of the script does not include installation of
packages. That happens after all the scripts terminated. The scripts
simply describe in what situation the OS should be.

It is possible to hook some functions in between (after, before)
installation of packages, or even between installation and
configuration.
71

72 73 74 75 76 77 78 79 80 81
URIs
----

Sometimes, an entity needs to be referenced somehow. Such entity may
live in the local filesystem or be on an external server.

These are the types of URIs supported:

* `http://`
* `https://`
82
* `file://`
83 84 85 86 87 88 89 90 91 92
* `data:`

The remote ones (`http` and `https`) may need verification of the
integrity of its content. The other are considered secure and don't
need any kind of verification.

The `data:` is slightly limited compared to what the standard (RFC
2397) allows. The media type and charset are irrelevant to the
updater and are therefore not supported.

Karel Koci's avatar
Karel Koci committed
93 94 95
NOTE: In previous versions there was an `internal:` URI but that one is no longer
available and can't be used.

96
Verification
97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117
~~~~~~~~~~~~

To make remote access secure we need to verify downloaded content. This
is relevant to `http://` and `https://` URIs.

Options relevant to `https://` URI::

ca;;
  An URI or table of URIs with trusted SSL certificate authorities, in PEM format.
  These URIs are not verified. `true` can be specified and in such case all system
  certificates are considered (generally all files in `/etc/ssl/certs`).
  If `false` or empty table is specified then CA verification is skipped. If not
  specified (or set to `nil`) it is inherited from the verification of script
  running the command. In default it is set to `true` so it verifies server
  against all installed CA certificates.
crl;;
  An URI or table of URIs with CRLs relevant to the server. Can be set to `false`
  or empty table and in such case CRL is not checked. If set to `nil` then its
  value is inherited from the verification of script running the command. In
  default it is set to `{}`.
ocsp;;
Karel Koci's avatar
Karel Koci committed
118
  `true` of `false` if you want or don't want to use OCSP (Online Certificate
119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139
  Status Protocol). If set to `nil` then value is inherited from verification of
  the script. Default value is `true`.

Options relevant to `http://` and `https://` URIs::

pubkey;;
  An URI or table of URIs with trusted public signature keys. These are not
  verified (therefore it is recommended to come from a already verified source ‒
  like `data:` or `file://` URI). If it is not specified (`nil`), it is inherited
  from the verification of the script running the command. If empty table is
  specified then no signature checking is done. Default value is `{}`.
sig;;
  URI where the signature of the resource lives. This one is not verified. If it
  is set to `nil`, it is constructed by adding `.sig` to the end of the verified
  URI. The option has effect only if `pubkey` is set so signature checking is
  done. In default it's set to `nil`.

URIs specified in these verification options are not verified (default values, not
inherited ones, are used). Because of that it is suggested to used only
trusted/secure URIs for that purpose. Suggested are `file://` and `data://`.

140 141 142
NOTE: Another option `verification` exist. It was originally used for verification
level specification but that is now replaced with `pubkey` and `ca` option
specific values. For backward compatibility it is silently ignored.
143

144 145 146 147
NOTE: For `ca` option there is also constant `system_cas` and for `crl` option
there is constant `no_crl`. These are obsoleted but are still defined.
`system_cas` is defined as `true` and `no_crl` is defined as `false`.
`system_cas` 
148

149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182
Dependency description
----------------------

Package dependencies are very important part of package maintenance.
Therefore, it is possible to describe them in the updater.

A dependency might be one of:

string::
  The string is parsed the same way as from the OpenWRT packages.
  Dependencies are separated by commas, each ``word'' meaning a single
  dependency.  The dependencies can also use versions with relational
  operators.  The version with the operator is in parentheses after
  the name of the package (eg `kernel (=version-number)`, `openssl
  (>=1.0.0)`). As an extension, the operator `~` may be used to denote
  a lua string pattern match on the version. All dependencies and
  version restrictions must be met.
package handle::
  A concrete package, represented by the result of the `Package`
  command may be used directly.
table::
  The table shall contain multiple sub-dependencies. Each one must be
  met. The sub-dependency may be of any type (string, other table,
  package handle, `Or()`, `Not()`).
`Not(string)`::
  This denotes that a single package described by the string must not
  be present on the system. This may be used if two packages ``fight''
  over the same file, or when they provide colliding services (both
  are DNS servers, for example).
`Or(dep, dep, ...)`::
  Each `dep` argument is a dependency of any type. The whole `Or()` is
  fulfilled when at least one of the argument is fulfilled. When
  multiple options are possible, the leftmost of them is preferred.

183 184 185
Available commands
------------------

Karel Koci's avatar
Karel Koci committed
186
Most of the commands has following format:
187 188 189

  Command("string", "string", {param = 1})

190 191 192
Script
~~~~~~

193
  Script("uri", { extra })
194

195
This command runs another script.
196

197
The uri provides the location of the script.
198

199
The second parameter is a table with extra information. It allows
200
fine-tuning the verification of URI and the way the script runs. The
201 202 203 204 205 206
current extra parameters are following.

security::
  Security level on which the script runs. It shall contain one of the
  above values. The name is case insensitive. It is not possible to
  raise the level, such attempt is reported as an error. If not
207
  specified, the level is deduced from the URI. If the URI is remote,
208
  it doesn't go above `remote`, otherwise it doesn't go above `local`.
209 210
optional::
  Set this to `true` to not fail if retrieval of this script fails.
211 212 213
sig::
pubkey::
ca::
214
crl::
215
ocsp::
216
  Options to verify the script integrity.
217

218 219
WARNING: Following format is now marked as obsolete and should not be used:
`Script("script-name", "uri", { extra })`
Karel Koci's avatar
Karel Koci committed
220

221 222 223
NOTE: There is also obsoleted extra option `ignore`. This should not be used and
any value set to it is effectively considered to be same as setting `optional` to
`true`.
Karel Koci's avatar
Karel Koci committed
224

225 226 227
Repository
~~~~~~~~~~

228
  Repository("repository-name", "uri", { extra })
229

230
This command introduces another repository of packages. The can
231
be used as a reference from other commands and is used in error
232 233
messages. Be aware that collision names are considered error and
such repositories are not considered.
234

235
The URI is expected to contain an OpenWRT repository in the format
236 237 238 239 240
produced by the buildroot.

Extra parameters are:

index::
241 242 243
  Overrides the URI at which the repository index lives and uses the
  main URI as the place where packages are downloaded from. Both
  gzipped and plain versions may be in the given URI. If the option is
244
  not listed, it is expected to be in `Packages.gz`.
245 246 247 248 249 250
priority::
  In case of a package being available in multiple directories, the
  package is taken from the repository with highest priority. In case
  of equality, the one introduced first wins. The default when the
  option is not specified is 50. The number must be an integer between
  0 and 100.
251 252 253 254 255
optional::
  Set this to `true` to not fail if it is not possible to receive repository for
  any reason or to parse it. This can be due to missing resource or invalid
  verification but in both cases this is not fatal for updater execution and it
  continues without this repository.
256 257 258
sig::
pubkey::
ca::
259
crl::
260
ocsp::
261
  Options to verify the index integrity.
262

263 264 265 266 267 268 269 270 271 272 273 274
NOTE: There is also obsoleted `subdirs` extra parameter. It was intended to be
used as a simple way to add multiple repositories at once. It had small trick
under its sleeve that it combined all those repositories under one name but that
effectively changes nothing. In new versions of Updater-ng this option is only
emulated and repository with name generated with following script is added
instead: `NAME-SUBDIR` where `NAME` is name of repository and `SUBDIR` is specific
`subdirs` values.

NOTE: There is also obsoleted extra option `ignore`. This should not be used and
any value set to it is effectively considered to be same as setting `optional` to
`true`.

275 276 277
Uninstall
~~~~~~~~~

Karel Koci's avatar
Karel Koci committed
278
  Uninstall("package", "package", { extra }, "package", "package", { extra })
279 280 281 282

This command takes multiple package names. It ensures none of the
packages is installed.

283 284
TIP: This is not needed most cases, since unneeded packages are removed
automatically.
285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300

Extra options modify the packages preceding them, but only up to the
previous extra options block. Therefore, the first two packages in the
example are modified by the first extra options block, the third and
fourth by the second block.

priority::
  In case of colliding requirements (the same package is required by
  an ``Install`` command or as a dependency of something), the
  requirement with the higher priority wins. In case of a draw, an
  error is reported. The priority defaults to 50 and must be between 0
  and 100.

Install
~~~~~~~

Karel Koci's avatar
Karel Koci committed
301
  Install("package", "package", { extra }, "package", "package", { extra })
302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338

This command is the opposite of `Uninstall`. It requires that a
package be present in the system.

The resolving of extra options acts the same as with `Uninstall`.

Available extra options:

priority::
  Acts the same as with `Uninstall`.
version::
  Limits the considered versions. This may be a single string or a
  table with multiple strings. If there are multiple, each is
  considered a condition and all must pass for a version to be
  accepted. Using of operators `<`, `<=`, `>`, `>=` is possible. Also,
  if the version is prefixed with `~`, it acts as a lua string match
  pattern against the version. So this would accept versions between 3
  and 7 and ignore the `.0` ones: `{ ">=3.0", "<=7.0", "~^%d+%.[1-9]%d*" }`.
  The default is no condition, therefore all versions available pass.
  From the versions that match and satisfy all dependency
  requirements, the one with highest version is chosen. In case when
  no available version matches, the currently installed version is
  also considered as a fallback.
repository::
  Usually, all repositories are searched according to their
  priorities. If you specify this option as a lua table, only the
  repositories listed here are searched, in the order in the table
  (ignoring the global priority).
reinstall::
  When set to any value, the package is re-installed even if the
  chosen version is already installed.
critical::
  If set to any value, the package and all its dependencies are
  considered critical. The updater will try harder to have it in a
  consistent state or be able to at least fix it without access to
  network. Other packages may stop working if the update is
  interrupted at the wrong time (for example by a power outage), but
Karel Koci's avatar
Karel Koci committed
339 340
  would be fixed by another finished updater run. For critical packages
  `priority` field is ignored.
341 342 343 344 345 346 347 348 349 350 351 352
optional::
  Set this to `true` to not fail if packages is not available from any configured
  repository. Be aware that this has implications if form of possible removed
  packages from system.

IMPORTANT: Package may be required to be installed or uninstalled multiple times
(for example by multiple scripts). All such requirements are tried to be met (eg.
by unifying the version options, etc).

NOTE: There is also obsoleted but still working option `ignore` which if set to
any boolean true value it is considered as if `optional` extra option would be
set to `true`.
353 354 355 356

Package
~~~~~~~

357
  Package("name", { extra })
358 359 360 361 362

This command allows amending a package from a repository. It allows
for adding dependencies (even negative or alternative dependencies).
It allows for specifying hooks ‒ functions that are run at specific
times. It also allows creation of virtual packages ‒ a package that
363
doesn't really exist, but can participates in the dependency computation.
364 365 366 367 368 369 370 371 372 373

A package may be amended multiple times. Each time the options are
merged into the package options.

The options are:

virtual::
  If set to any value, the package is virtual. If a real package of
  the same name exists, an error is reported.
deps::
374 375 376
  Additional dependencies for the package. The dependencies are merged
  together as if all the sources were put into a table (eg. all of
  them must be fulfilled). There's no way to remove dependencies.
377 378
order_after::
order_before::
379 380 381 382 383 384
  Usually, all dependencies of a package are installed before the
  package. Sometimes, it may be desirable to break this order and
  these options allow that. Both of them list packages after or before
  which the current package shall be installed, in a table. This
  allows breaking dependency cycles. These options are mere hint, the
  updater may decide to not follow them if it is not possible to
385 386
  satisfy. Note that this has effect only on running the pre_* and
  post_* scripts and hooks, since all the files of all updated
387
  packages are merged into the system together.
388 389
pre_*::
post_*::
390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411
  A hook to be run after or before a step. The value may be a single
  function or a table of functions. All the functions (from the table
  or merged from multiple `Package` commands) are run, in unspecified
  order. TODO: List the steps and what commands may be used inside the
  functions.
reboot::
  A reboot is needed when installing the package. The reboot is
  scheduled according to the value.
  delayed;;
    The package needs a reboot for the new version to take effect, but
    the old version works, so it may be delayed for arbitrary amount
    of time.
  finished;;
    The reboot needs to be done once the update is finished. It is
    because the old version no longer works as expected. The whole
    update may be delayed because the need of this update, so the
    update happens at a time convenient to the user.
  immediate;;
    The reboot needs to be done just after the package is set up. This
    may be needed when the old version would prevent the rest of the
    update from happening.
replan::
412 413 414 415 416 417 418 419 420 421 422 423
  The package has an effect on the updater itself. Therefore, updater
  have to run planning again. When this happens is according to value.
  finished;;
    Replan is done after whole initial update was performed. Use this option
    if your package changes some setting that updater reads. It can be some
	uci config or even some additional updater configuration (usable for
    example if you want to add repository by installing package).
  immediate;;
    When this is set it interrupts the update after this package is set
    up. This causes updater to run planning again as soon as the change is made.
    This should be used only for packages containing updater itself or
    closely related packages.
424 425 426 427 428 429 430
abi_change::
  The package changed its ABI (or some other interface) and some other
  packages need to be reinstalled. If this is set to `true` and the
  package is installed or updated, all packages that depend on it are
  reinstalled. If it is set to a table, it lists packages that need to
  be reinstalled. When merging `true` to a table, `true` is considered
  to be the list of packages depending on this one.
431 432 433 434 435
abi_change_deep::
  Same as abi_change, but also reinstall packages that depends indirectly on
  package that changed its ABI. That means if some package is reinstalled because
  of change of ABI, all packages that depends on it are also reinstalled and so
  on.
436 437 438 439

NOTE: Originally there was also option `ignore` that allowed pass for different
problems but most of those were not working and usage of them was questionable.
This options is now considered as obsolete and is ignored.
Michal 'vorner' Vaner's avatar
Michal 'vorner' Vaner committed
440

441 442 443
Export and Unexport
~~~~~~~~~~~~~~~~~~~

Karel Koci's avatar
Karel Koci committed
444 445
  Export("variable")
  Unexport("variable")
446 447 448

These commands are described in section `Export variables to Script`.

449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466
version_cmp and version_match
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  version_cmp("version1", "version2")
  version_match("version", "version_relation")

Both of these functions implement way to compare versions of package.

`version_cmp` compares two version strings and returns `-1`, `0`, or `1`
if version in first argument is less, equal, or greater than version in second
argument respectively.

`version_match` allows you to use same syntax as in case of string dependency.
So you will pass version string as first argument and then string with version
relation as second argument (for example:
`version_match(installed['pkg'].version, '>=1.0.0')`)
It returns true if version matches given version relation, otherwise false.

467 468 469
Logging
~~~~~~~

Karel Koci's avatar
Karel Koci committed
470 471 472 473
  DBG("debug text")
  INFO("information text")
  WARN("warning text")
  ERROR("error text")
474
  DIE("error text")
475 476

These commands allows printing of messages for their corresponding
477 478
verbosity levels. On top of that `DIE` command also results to
updater failure.
479

480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515
Access files
~~~~~~~~~~~~

On top of standard `io` and `file` updater also defines some of its own functions
for files access.

  ls(path)
  stat(path)
  lstat(path)

All these functions have to be called with string argument containing path. Note
that you should prepend all paths with `root_dir` variable to support off root
execution.

ls::
  This functions returns table with file names under the given path (acts like
  shell's `ls -a`, ommiting `.` and `..` entries). If given path is not directory
  or doesn't exists then error is raised. Value in table for each key is the type
  of file, which may be:
    b;; A block device
    c;; A character device
    d;; A directory
    f;; A named pipe
    l;; A symbolic link
    r;; A regular file
    s;; A unix-domain socket
    ?;; Failed to determine the type
stat::
  Statistics about the given file. If the file does not exist, it returns nothing.
  Otherwise, the file type is returned (see the types of `ls`). The second result
  is the permissions of the file, in the imitation of shell's `ls -l`, like
  `rwxr-x---`.
lstat::
  Same as `stat` except the `lstat` behaviour is preferred. (eg. provides info
  about symbolic link if it is a link, instead of the target).

516 517
Predefined variables
--------------------
518 519 520 521 522 523 524

There are several global variables. These are set anew for each script
run, so one script can't damage them for another. Modifying them has
no effect on the updater's behaviour, unless specifically mentioned.
Note that some of the tables might be generated on demand by
meta-table events, making it impossible to list keys.

525 526 527 528 529 530
root_dir
~~~~~~~~

Root directory specified from command line or `/` if no such option
was specified. Use this if you are accessing some files.

531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551
self_version
~~~~~~~~~~~~

String containing version of updater.

language_version
~~~~~~~~~~~~~~~~

Number signaling version of updater configuration language used. This
is always `1` for language described in this document.

features
~~~~~~~~

Set of features current updater supports. You can check for feature
this way: `features['feature']`. These are currently available features:

priorities::
  Updater handles priorities between multiple requests for same package.
provides::
  Updater supports `Provides` control field.
Karel Koci's avatar
Karel Koci committed
552 553
conflicts::
  Updater supports `Conflicts` control field.
554 555
abi_change::
  Updater can handle and propagate ABI change.
556 557
abi_change_deep::
  Updater can handle and propagate deep ABI change.
Karel Koci's avatar
Karel Koci committed
558
replan_string::
559 560
  Updater expects replan to be a string (if this feature isn't set than it's
  expected to be only boolean).
561 562 563
relative_uri::
  Updater supports relative URI where URI is deduced relative to script in which
  it was defined in.
564 565 566
no_returns::
  Functions such as `Repository` and `Package` no longer return handler that can
  be used in other calls.
567

568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585
installed
~~~~~~~~~

This is a table of installed packages. The keys are package names and
values are tables with following keys:

version::
  The installed version.
files::
  Files belonging to the package (a table).
configs::
  Configuration files belonging to the package (a table).
repository::
  Name of the repository it has been installed from. It may be missing
  in case it is a package provided outside of a repository. Note that
  the name corresponds to the time the package has been installed and
  that repository may be unavailable now or the name represent a
  different repository.
586
install_time::
587 588 589 590 591 592 593
  Unix timestamp specifying when the package has been installed, in
  UTC.

The top-level table is instantiated (not generated through
meta-tables), therefore it is possible to get the list of installed
packages.

594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619
os_release
~~~~~~~~~~

This is table with parsed content of os-release file. Path to this file is
`etc/os-release` but relative to target root. This means that if you are running
updater on root file system that is not current root then values in this table are
for target not for host system.

This is normal table and you can iterate trough it using `pairs` or you can
directly access specific value by indexing it. List of standard options can be
found https://www.freedesktop.org/software/systemd/man/os-release.html[here].

The most interesting value is `os_release.VERSION` as this contains current system
release version.

This table can be empty if there was no `os-release` file.

host_os_release
~~~~~~~~~~~~~~~

This is table with parsed content of os-release file for host system. Source file
for this is always `/etc/os-release`. See variable os_release for example usage
and expected content.

Table can be empty if there was no `/etc/os-release`.

620 621 622 623 624 625 626
Export variables to Script
--------------------------

For security reasons individual scripts doesn't share variables. But it's
sometimes beneficial to have variable that can be read by sub-script. Such
variable is so called exported.

Karel Koci's avatar
Karel Koci committed
627 628
  Export('variable')
  Unexport('variable')
629 630 631 632 633 634 635

To export variable you have to call `Export` function with name of variable as
argument. To revert that you have to call `Unexport` function with same argument.

Be aware that changes done in sub-script to exported variables are not propagated
back to original script. But it's propagated to script executed from sub-script.

636 637
Hooks
-----
638

639
As the hooks are run after all the dependencies have been resolved and
640 641
plan has been made, it is no longer possible to call the `Repository`,
`Package`, `Install` and `Uninstall` commands.
642

643 644 645 646 647 648
The only parameter of the hook is a package object. It is a table that
has all the meaningful options from the `Package` and `Install`
commands and the ones from `installed` variable. However, with options
listing multiple possibilities, only the one chosen is kept. Also,
additional `name` option is included.

649 650 651 652 653 654 655 656
Available libraries and standard functions
------------------------------------------

In addition to the functions listed above, following functions and
libraries are made available. They are available in the security level
listed and in all higher levels.

Restricted::
657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677
  * `table` library
  * `string` library
  * `math` library
  * `assert`
  * `error`
  * `ipairs`
  * `next`
  * `pairs`
  * `pcall`
  * `select`
  * `tonumber`
  * `tostring`
  * `type`
  * `unpack`
  * `xpcall`
  * `DBG`
  * `INFO`
  * `WARN`
  * `ERROR`
  * `version_cmp`
  * `version_match`
678
Local::
679
  * `uci` library
680 681 682 683 684
  * `io`
  * `file`
  * `ls`
  * `stat`
  * `lstat`
685
Full::
686
  * The whole lua library