updater issueshttps://gitlab.nic.cz/turris/updater/updater/-/issues2020-11-12T02:24:15+01:00https://gitlab.nic.cz/turris/updater/updater/-/issues/1Integrate the lua interpreter2020-11-12T02:24:15+01:00Ghost UserIntegrate the lua interpreterMake sure the opkg-trans builds with a lua interpreter, the interpreter is properly initiated, etc. We need a way to inject functions to lua.Make sure the opkg-trans builds with a lua interpreter, the interpreter is properly initiated, etc. We need a way to inject functions to lua.https://gitlab.nic.cz/turris/updater/updater/-/issues/2Parse current package state2020-11-12T02:24:16+01:00Ghost UserParse current package stateParse the info in /usr/lib/opkg and provide it as some data structures in lua. Make sure it is easily tested.
- [x] Depends on #1Parse the info in /usr/lib/opkg and provide it as some data structures in lua. Make sure it is easily tested.
- [x] Depends on #1https://gitlab.nic.cz/turris/updater/updater/-/issues/3Event loop2020-11-12T02:24:16+01:00Ghost UserEvent loop - Integrate libevent
- Handle registration of children and call callbacks for them (can we reuse the libevent's events for that?)
- Some kind of „wait“ mode ‒ list events that must stop being pending or active to stop, while still exe... - Integrate libevent
- Handle registration of children and call callbacks for them (can we reuse the libevent's events for that?)
- Some kind of „wait“ mode ‒ list events that must stop being pending or active to stop, while still executing the rest of eventshttps://gitlab.nic.cz/turris/updater/updater/-/issues/4Process manager2020-11-12T02:24:16+01:00Ghost UserProcess managerImplement a process manager, inspired by perl's AnyEvent::Util::run_command (but simpler). It needs to be able to run the command provided, specify callbacks or buffers for its stdio and run a callback when all the IO is done and the com...Implement a process manager, inspired by perl's AnyEvent::Util::run_command (but simpler). It needs to be able to run the command provided, specify callbacks or buffers for its stdio and run a callback when all the IO is done and the command terminated.
Provide some interface to lua.
- [x] Depends on #1
- [x] Depends on #3https://gitlab.nic.cz/turris/updater/updater/-/issues/5Unpacking packages to a temporary location2020-11-12T02:24:16+01:00Ghost UserUnpacking packages to a temporary locationHave a mechanism to unpack the content of a given package into a given place and extract the control files somewhere.
Consider feeding it with a buffer instead of a file name.
- [x] Depends on #4Have a mechanism to unpack the content of a given package into a given place and extract the control files somewhere.
Consider feeding it with a buffer instead of a file name.
- [x] Depends on #4https://gitlab.nic.cz/turris/updater/updater/-/issues/6Merging files to live system2019-05-06T17:46:51+02:00Ghost UserMerging files to live systemMerge requests to install files from the unpacked packages (in the temporary location) and requests to remove files by updated or removed packages into live system. This doesn't yet contain modifying the package state database, but it sh...Merge requests to install files from the unpacked packages (in the temporary location) and requests to remove files by updated or removed packages into live system. This doesn't yet contain modifying the package state database, but it should check for package collisions from other packages (either already installed or being installed now) and correctly identifying what needs to be removed.
- [x] Depends on #5
- [ ] Related to #8https://gitlab.nic.cz/turris/updater/updater/-/issues/7Running pre/post-install/remove scripts of packages2020-11-12T02:24:16+01:00Ghost UserRunning pre/post-install/remove scripts of packagesRun the necessary scripts. Decide if we want to run them before and after the whole merging of files #6 , or during. But we probably don't want to abort the merging in the middle because of a failed script.
- [x] Depends on #6
- [x] ...Run the necessary scripts. Decide if we want to run them before and after the whole merging of files #6 , or during. But we probably don't want to abort the merging in the middle because of a failed script.
- [x] Depends on #6
- [x] Depends on #8https://gitlab.nic.cz/turris/updater/updater/-/issues/8Saving package state2020-11-12T02:24:16+01:00Ghost UserSaving package stateBe able to store the package state and use it after the installation (or prepare it in the temporary location before merging?).
- [x] Related to #6
- [x] Depends on #2Be able to store the package state and use it after the installation (or prepare it in the temporary location before merging?).
- [x] Related to #6
- [x] Depends on #2https://gitlab.nic.cz/turris/updater/updater/-/issues/9Locking2020-11-12T02:24:11+01:00Ghost UserLockingPerform some kind of locking (on the journal? On the package state? See what opkg locks, if anything?) to prevent parallel runs. Decide if we want to wait (with timeout?) or abort.Perform some kind of locking (on the journal? On the package state? See what opkg locks, if anything?) to prevent parallel runs. Decide if we want to wait (with timeout?) or abort.https://gitlab.nic.cz/turris/updater/updater/-/issues/10Journal manipulation2020-11-12T02:24:15+01:00Ghost UserJournal manipulationBe able to read and write the journal. Writing needs to be atomic in some way, reading might need to handle damaged or incomplete journal. Decide on the format, but don't act on what is being read.Be able to read and write the journal. Writing needs to be atomic in some way, reading might need to handle damaged or incomplete journal. Decide on the format, but don't act on what is being read.https://gitlab.nic.cz/turris/updater/updater/-/issues/11Using the journal through installation2020-11-12T02:24:15+01:00Ghost UserUsing the journal through installationUse the journal when we install or remove packages ‒ write what is being done, etc, to ensure we can recover.
- [x] Depends on #10Use the journal when we install or remove packages ‒ write what is being done, etc, to ensure we can recover.
- [x] Depends on #10https://gitlab.nic.cz/turris/updater/updater/-/issues/12Journal recovery2020-11-12T02:24:16+01:00Ghost UserJournal recoveryBe able to recover installation when an incomplete journal is found. Implement journal abortion in possible cases (when merging hasn't started yet).
- [x] Depends on #11Be able to recover installation when an incomplete journal is found. Implement journal abortion in possible cases (when merging hasn't started yet).
- [x] Depends on #11https://gitlab.nic.cz/turris/updater/updater/-/issues/13Integrate the backend in the current updater2020-11-12T02:24:16+01:00Ghost UserIntegrate the backend in the current updaterGather all the packages to install or remove and perform the whole removal/installation at once by the opkg-trans backend.
- [x] Depends on #35 (Field test)Gather all the packages to install or remove and perform the whole removal/installation at once by the opkg-trans backend.
- [x] Depends on #35 (Field test)https://gitlab.nic.cz/turris/updater/updater/-/issues/14Transaction backend for the old updater2019-05-06T17:46:55+02:00Ghost UserTransaction backend for the old updaterImplement the backend part of the new updater and replace opkg with this in the old updater.
- [x] Subtask #1
- [x] Subtask #2
- [x] Subtask #3
- [x] Subtask #4
- [x] Subtask #5
- [x] Subtask #6
- [x] Subtask #7
- [x] Subtask #8...Implement the backend part of the new updater and replace opkg with this in the old updater.
- [x] Subtask #1
- [x] Subtask #2
- [x] Subtask #3
- [x] Subtask #4
- [x] Subtask #5
- [x] Subtask #6
- [x] Subtask #7
- [x] Subtask #8
- [x] Subtask #9
- [x] Subtask #10
- [x] Subtask #11
- [x] Subtask #12
- [x] Subtask #13
- [x] Subtask #20
- [x] Subtask #25
- [x] Subtask #26
- [x] Subtask #27
- [x] Subtask #32 (Logging configuration)
- [x] Subtask #35 (Field test)https://gitlab.nic.cz/turris/updater/updater/-/issues/15LibC replacement support2019-05-06T17:47:25+02:00Ghost UserLibC replacement supportWe'll need to replace libc. Doing so is little bit complex, so we need some support to do that (static linking, special handling, journal, etc).
As it is currently unclear, what is enough to switch libc, here's what we may need to try:
...We'll need to replace libc. Doing so is little bit complex, so we need some support to do that (static linking, special handling, journal, etc).
As it is currently unclear, what is enough to switch libc, here's what we may need to try:
- „Just“ reinstall all the packages in the system. We probably need to push new-enough updater into the 2.* generation and then start separate generation 3. The generation 3 would be linked with musl, while 2 would be with uclibc. The new version would know how to do „whole system reinstall“ when seen on libc package.
- Statically link the updater so it can start in an intermediate state. This might be needed when switching the updater. Or, it may be needed during a recovery, but as the init scripts wouldn't run at that time, it is unlikely that would help.
- Embed statically-linked busybox inside the opkg-trans binary and use that one to run commands, as an extension of the above.
- Thorough testing.
These are the tasks that would be needed to accomplish it:
- [x] Depends on #14 (opkg-trans backend)
- [x] Subtask turris/openwrt#9 (Turris packages with musl)
- [x] Subtask #29 (Reinstall everything flag)
- [x] Subtask #31 (Testing the migration)
- [x] Subtask turris/openwrt#10 (static libraries for dependencies, maybe not needed
- [x] Subtask #30 (static linking), maybe not needed
- [x] Depends on #148 (Replace curl command with libcurl)
- [x] Depends on #149 ( Integrate busybox to updater)
https://gitlab.nic.cz/turris/updater/updater/-/issues/16Replacement for the old updater2019-05-06T17:46:55+02:00Ghost UserReplacement for the old updaterBring the updater to a state where it can read the language, resolve dependencies, and install packages. In this phase we could replace the old updater, but we wouldn't have all the new fancy stuff, like complex dependencies.
- [x] Dep...Bring the updater to a state where it can read the language, resolve dependencies, and install packages. In this phase we could replace the old updater, but we wouldn't have all the new fancy stuff, like complex dependencies.
- [x] Depends on #14
- Language support:
- [x] Subtask #36 (Sandboxes)
- [x] Subtask #37 (Morphers)
- [x] Subtask #50 (Include command)
- [x] Subtask #97 (Errors from sub-scripts)
- [x] Subtask #76 (Allow missing)
- [x] Subtask #86 (Unkown parameters)
- [x] Subtask #91 (Complex deps)
- [x] Subtask #92 (Flags)
- [x] Subtask #93 (State variables)
- Data sources:
- [x] Subtask #38 (Download event)
- [x] Subtask #39 (URI manager)
- [x] Subtask #59 (Resource verification)
- [x] Subtask #60 (internal: uri scheme)
- [x] Subtask #103 (Restricted security level)
- Situation description:
- [x] Subtask #40 (Package object)
- [x] Subtask #41 (Repository object)
- [x] Subtask #42 (Installation request)
- Request post-processing
- [x] Subtask #57 (Download repositories)
- [x] Subtask #58 (Get table of all packages)
- Dependency tracking:
- [x] Subtask #43 (DFS through dependencies)
- [x] Subtask #44 (Compare list of desired and installed packages)
- [x] Subtask #45 (Feed the package list into the transaction backend)
- [x] Subtask #109 (The „provides“ header)
- UI
- [x] Subtask #46 (The updater binary)
- [ ] Subtask #47 (Command to remove or add a package)
- [x] Subtask #48 (Emulation of opkg)
- [x] Subtask #49 (Emulation of the old updater)
- [x] Subtask #67 (Post the log of installed packages)
- [x] Subtask #69 (Better error reporting)
- [x] Subtask #77 (Nuci integration)
- [x] Subtask #80 (Collision reporting)
- [x] Subtask #83 (CRL error)
- [x] Subtask #84 (Missing repository error)
- Misc
- [x] Subtask #61 (Hide context)
- [x] Subtask #68 (Access to system libraries)
- [x] Subtask #62 (Replan)
- [x] Subtask #63 (Reinstall)
- [x] Subtask #96 (Late reinstall)
- [x] Subtask #64 (Reboot)
- [x] Subtask #65 (TODO/FIXME)
- [x] Subtask #66 (Deploy)
- [x] Subtask #72 (Dont fail on prerm)
- [x] Subtask #74 (Lock for the whole run)
- [x] Subtask #75 (Config overwrining)
- [x] Subtask #81 (Blocking write)
- [x] Subtask #95 (Lockup on poll)
- [x] Subtask #87 (Auto-run resilience)
- [x] Subtask #88 (User-list definitions)https://gitlab.nic.cz/turris/updater/updater/-/issues/17Full support of the requirements and language2019-05-06T17:47:26+02:00Ghost UserFull support of the requirements and languageAll the rest of language not covered under #16 comes here. It includes all that conflict resolution, alternative dependencies, etc.
- [x] Subtask #151 (Repository integrity error handling)
- [ ] Subtask #137 (In plan ordering support op...All the rest of language not covered under #16 comes here. It includes all that conflict resolution, alternative dependencies, etc.
- [x] Subtask #151 (Repository integrity error handling)
- [ ] Subtask #137 (In plan ordering support options order_after and order_before)
- [x] Subtask #150 (Package ignore installation errors)
- [x] Subtask #199 (Add support for hooks as described in Language design)
TODO this is not full list.
https://gitlab.nic.cz/turris/updater/updater/-/issues/19Updater NG2019-05-06T17:46:55+02:00Ghost UserUpdater NGThe current hair ball of shell scripts is slowly but surely reaching its limits. Consider some newer approach that does real dependency tracking and such.
- [x] Subtask #14 (Transactional backend for the old updater)
- [x] Subtask #...The current hair ball of shell scripts is slowly but surely reaching its limits. Consider some newer approach that does real dependency tracking and such.
- [x] Subtask #14 (Transactional backend for the old updater)
- [x] Subtask #1 (Integrate lua interpreter)
- [x] Subtask #2 (Parse current package state)
- [x] Subtask #3 (Event loop)
- [x] Subtask #20 (Logging support)
- [x] Subtask #4 (Process manager)
- [x] Subtask #5 (Unpacking to temporary location)
- [x] Subtask #6 (Merging files to live system)
- [x] Subtask #7 (Running pre/post-install/remove scripts)
- [x] Subtask #8 (Saving package state)
- [x] Subtask #9 (Locking)
- [x] Subtask #10 (Journal manipulation)
- [x] Subtask #11 (Using journal through the installation)
- [x] Subtask #12 (Journal recovery)
- [x] Subtask #13 (Integrate the backend in the old updater)
- [x] Subtask #25 (Integrate `transaction.process` into opkg-trans)
- [x] Subtask #26 (Config file handling)
- [x] Subtask #27 (System level tests)
- [x] Subtask #32 (Logging configuration)
- [x] Subtask #35 (Field test)
- [ ] Subtask #15 (LibC replacement support)
- [ ] Subtask turris/openwrt#9 (Turris packages with musl)
- [x] Subtask #29 (Reinstall everything flag)
- [ ] Subtask #31 (Testing the migration)
- [ ] Subtask turris/openwrt#10 (static libraries for dependencies, maybe not needed
- [ ] Subtask #30 (static linking), maybe not needed
- [x] Depends on #148 (Replace curl command with libcurl)
- [x] Depends on #149 ( Integrate busybox to updater)
- [x] Subtask #16 (Replacement for the old updater)
- Language support:
- [x] Subtask #36 (Sandboxes)
- [x] Subtask #37 (Morphers)
- [x] Subtask #50 (Include command)
- [x] Subtask #76 (Allow missing)
- [x] Subtask #86 (Unknown parameters)
- [x] Subtask #91 (Complex deps)
- [x] Subtask #92 (Flags)
- [x] Subtask #93 (State variables)
- [x] Subtask #97 (Errors from sub-scripts)
- Data sources:
- [x] Subtask #38 (Download event)
- [x] Subtask #39 (URI manager)
- [x] Subtask #59 (Verification of resources)
- [x] Subtask #60 (internal: schema)
- [x] Subtask #103 (Restricted security level)
- Situation description:
- [x] Subtask #40 (Package object)
- [x] Subtask #41 (Repository object)
- [x] Subtask #42 (Installation request)
- Postprocess info from configs:
- [x] Subtask #57 (Download repositories)
- [x] Subtask #58 (Get table of available packages)
- Dependency tracking:
- [x] Subtask #43 (DFS through dependencies)
- [x] Subtask #44 (Compare list of desired and installed packages)
- [x] Subtask #45 (Feed the package list into the transaction backend)
- [x] Subtask #109 (The „provides“ header)
- UI
- [x] Subtask #46 (The updater binary)
- [ ] Subtask #47 (Command to remove or add a package)
- [x] Subtask #48 (Emulation of opkg)
- [x] Subtask #49 (Emulation of the old updater)
- [x] Subtask #67 (Send logs of transaction to user)
- [x] Subtask #69 (Better error reporting)
- [x] Subtask #77 (Nuci integration)
- [x] Subtask #80 (List collisions)
- [x] Subtask #83 (CRL error)
- [x] Subtask #84 (Missing repo error)
- [ ] Subtask #136 (Describe why critical requests failed in error message)
- Misc
- [x] Subtask #61 (Hide context)
- [x] Subtask #68 (Access to system libraries)
- [x] Subtask #62 (Replan)
- [x] Subtask #63 (Reinstall)
- [x] Subtask #96 (Later reinstall)
- [x] Subtask #64 (Reboot)
- [x] Subtask #65 (TODO/FIXME)
- [x] Subtask #66 (Deploy)
- [x] Subtask #72 (Don't fail on prerm)
- [x] Subtask #74 (Lock during the whole run)
- [x] Subtask #75 (Config overwriting)
- [x] Subtask #81 (Blocking write)
- [x] Subtask #87 (Auto-run resilience)
- [x] Subtask #88 (Generator of the configs)
- [x] Subtask #89 (Userlist definitions)
- [x] Subtask #95 (Lockup on poll)
- [x] Subtask #18 (Rules for dependency calculation)
- [x] Subtask #122 (SAT structure, stage 1)
- [x] Subtask #124 (Satisfiable requests)
- [x] Subtask #125 (Eliminate unneeded packages)
- [x] Subtask #127 (Order packages according to dependencies)
- [x] Subtask #128 (Produce list of tasks to execute)
- [x] Subtask #123 (SAT structure, stage 2)
- [x] Subtask #126 (Alternative elimination)
- [ ] Subtask #17 (Full support of the requirements)https://gitlab.nic.cz/turris/updater/updater/-/issues/20Logging support2020-11-12T02:24:16+01:00Ghost UserLogging supportProvide at least basic logging functions, eg:
- die
- error
- warn
- dbg
Provide lua interface as well.
It doesn't have to be configurable or output to multiple locations (yet). We just need a way to print stuff from the code.
-...Provide at least basic logging functions, eg:
- die
- error
- warn
- dbg
Provide lua interface as well.
It doesn't have to be configurable or output to multiple locations (yet). We just need a way to print stuff from the code.
- [x] Depends on #1https://gitlab.nic.cz/turris/updater/updater/-/issues/21Restart updater even in offline mode2019-05-06T17:46:56+02:00Michal HruseckyRestart updater even in offline modeMake it possible to restart updater even if it is run after boot in offline mode. Implementation in merge request !14 Make it possible to restart updater even if it is run after boot in offline mode. Implementation in merge request !14 Michal HruseckyMichal Hrusecky