Device in exclamation mark state
Succesfully paired mox (paired to omnia) state is "exclamation mark".
Also wifi setup is inaccessible with the same state.
Nevertheless, wifi on mox is configured same as on omnia, so it is more matter of wrong detection of Mox state.
Probable cause
Lost connection to paired device
Affected software and versions
TOS 4.0.5 (HBS) through 5.1 (HBL) on Omnia, TOS 4.0.5 on Mox
- turris/turris-netboot #4Turris OS 5.2.0
Activity
added Bug label
added High label
changed milestone to %Turris OS 5.0
assigned to @mhrusecky
Should be fixed by these two PRs: https://gitlab.labs.nic.cz/turris/turris-netboot/-/merge_requests/4 https://gitlab.labs.nic.cz/turris/turris-os-packages/-/merge_requests/289
Can you please @mmatejek check it?
Not really, it is fixing an issue where MOX wasn't booting at all, but afterwards it worked for @jhoracek even with the GUI :-/
-
Removed label Review. :(
added Review label
@bbodnar, could you please take a look?
I am sorry for writing the answer here, I do not know how to sent you a PM over gitlab. I used
switch-branch hbkon my OMNIA, but my question was how to run this on netbooted MOX. I thought that netbooted MOX is not accessible via LUCI nor via ssh. Which system is essentially running there if there is no SD card in it (in netbooted MOX)? Or do I miss something?
-
There exists a possibility how to change Turris OS version on netbooted device, but it is not tested by our tests. The rootfs which downloads it points to HBS and this is officially supported. If you want still do to that, you can try to modify file
/usr/bin/netboot-manageron the router from which the devices boots. And with the next update ofturris-netboot-toolspackage, the contents will be overwritten.Edited by Josef Schlehofer I did that, I copied the
/usr/bin/netboot-managerto/usr/bin/netboot-manager-hbk, replacehbswithhbkin the/usr/bin/netboot-manager-hbk, chnaged the permissions to0755and ran/usr/bin/netboot-manager-hbk regen -f. It went trough ok (again with the warning messages). I rebooted the MOX. But the problem with exclamation mark still persists.$ netboot-manager-hbk regen -f Regenerating configuration... Getting new rootfs... Downloading 'https://repo.turris.cz/hbk/netboot/mox-netboot-latest.tar.gz' Connecting to 2001:1488:ac15:ff80::69:443 Writing to '/srv/turris-netboot/rootfs/rootfs-new.tar.gz' /srv/turris-netboot/ 100% |*******************************| 44553k 0:00:00 ETA Download completed (45622572 bytes) Downloading 'https://repo.turris.cz/hbk/netboot/mox-netboot-latest.tar.gz.sha256' Connecting to 2001:1488:ac15:ff80::69:443 Writing to '/srv/turris-netboot/rootfs/rootfs-new.tar.gz.sha256' /srv/turris-netboot/ 100% |*******************************| 98 0:00:00 ETA Download completed (98 bytes) Downloading 'https://repo.turris.cz/hbk/netboot/mox-netboot-latest.tar.gz.sig' Connecting to 2001:1488:ac15:ff80::69:443 Writing to '/srv/turris-netboot/rootfs/rootfs-new.tar.gz.sig' /srv/turris-netboot/ 100% |*******************************| 151 0:00:00 ETA Download completed (151 bytes) rootfs-new.tar.gz: OK OK mox.its:8.18-23.11: Warning (unit_address_vs_reg): /images/kernel@0: node has a unit name, but no reg property mox.its:17.20-19.15: Warning (unit_address_vs_reg): /images/kernel@0/hash@1: node has a unit name, but no reg property mox.its:20.20-22.15: Warning (unit_address_vs_reg): /images/kernel@0/hash@2: node has a unit name, but no reg property mox.its:24.19-37.11: Warning (unit_address_vs_reg): /images/ramdisk@0: node has a unit name, but no reg property mox.its:31.20-33.15: Warning (unit_address_vs_reg): /images/ramdisk@0/hash@1: node has a unit name, but no reg property mox.its:34.20-36.15: Warning (unit_address_vs_reg): /images/ramdisk@0/hash@2: node has a unit name, but no reg property mox.its:38.15-50.11: Warning (unit_address_vs_reg): /images/fdt@0: node has a unit name, but no reg property mox.its:44.20-46.15: Warning (unit_address_vs_reg): /images/fdt@0/hash@1: node has a unit name, but no reg property mox.its:47.20-49.15: Warning (unit_address_vs_reg): /images/fdt@0/hash@2: node has a unit name, but no reg property mox.its:55.16-60.11: Warning (unit_address_vs_reg): /configurations/conf@1: node has a unit name, but no reg property FIT description: Netboot image with single Linux kernel and FDT blob Created: Thu Apr 2 13:55:39 2020 Image 0 (kernel@0) Description: Linux kernel without Created: Thu Apr 2 13:55:39 2020 Type: Kernel Image (no loading done) Compression: uncompressed Data Size: 10244104 Bytes = 10004.01 KiB = 9.77 MiB Hash algo: crc32 Hash value: 8762345b Hash algo: sha1 Hash value: a200f34b7e8c4c224589f7d3c32fec76bc59875e Image 1 (ramdisk@0) Description: initramfs Created: Thu Apr 2 13:55:39 2020 Type: RAMDisk Image Compression: uncompressed Data Size: 6974464 Bytes = 6811.00 KiB = 6.65 MiB Architecture: AArch64 OS: Linux Load Address: unavailable Entry Point: unavailable Hash algo: crc32 Hash value: 0c2397a5 Hash algo: sha1 Hash value: 2a4f2d590bbe83a690dff2c325d9e2955a912f19 Image 2 (fdt@0) Description: Flattened Device Tree blob Created: Thu Apr 2 13:55:39 2020 Type: Flat Device Tree Compression: uncompressed Data Size: 19412 Bytes = 18.96 KiB = 0.02 MiB Architecture: AArch64 Hash algo: crc32 Hash value: da77bf90 Hash algo: sha1 Hash value: 27d0d7924d67953d33a3c5065444c8aee9f2b96c Default Configuration: 'conf@1' Configuration 0 (conf@1) Description: Turris MOX netboot image Kernel: kernel@0 Init Ramdisk: ramdisk@0 FDT: fdt@0 Signing kernels...
removed Review label
@mmatejek, I've finally reproduced it. But I didn't get any error message in the browser console. Moreover, I got similar behavior in reForis. I'm pretty sure that the problem is on the backend side.
I'm sorry I didn't catch any error message, now I can't boot MOX even from flash drive for some reason :D I'll try to debug in one more time later and let you know.
Edited by Bogdan Bodnar-
Ok, I got it worked again and what I got for some time of debugging:
It seems that MOX sends a message without problems:
Apr 2 12:44:51 turris foris-controller[8508]: DEBUG:foris_client.buses.mqtt:Msg recieved for 'foris-controller/0000000D30000165/notification/remote/action/advertize' (msg=b'{"module": "remote", "action": "advertize", "kind": "notification", "data": {"state": "running", "id": "0000000D30000165", "hostname": "turris", "netboot": "booted", "modules": [{"name": "about", "version": "1.0.9"}, {"name": "dns", "version": "1.0.9"}, {"name": "guest", "version": "1.0.9"}, {"name": "lan", "version": "1.0.9"}, {"name": "maintain", "version": "1.0.9"}, {"name": "networks", "version": "1.0.9"}, {"name": "password", "version": "1.0.9"}, {"name": "remote", "version": "1.0.9"}, {"name": "router_notifications", "version": "1.0.9"}, {"name": "time", "version": "1.0.9"}, {"name": "updater", "version": "1.0.9"}, {"name": "wan", "version": "1.0.9"}, {"name": "web", "version": "1.0.9"}, {"name": "wifi", "version": "1.0.9"}], "working_replies": []}}') Apr 2 12:44:51 turris foris-controller[8508]: DEBUG:foris_client.buses.mqtt:Msg for 'foris-controller/0000000D30000165/notification/remote/action/advertize' was processedBut something suspicious happens on omnia:
Apr 2 12:47:30 turris foris-controller[11094]: DEBUG:foris_client.buses.mqtt:Msg recieved for 'foris-controller/0000000B00009CF7/reply/ef52ae9d-13eb-4f72-a30b-10bc6bd753df' (msg=b'') Apr 2 12:47:30 turris foris-controller[11094]: DEBUG:foris_client.buses.mqtt:Message id not found. (probably it is expired or doesn't belong to this client) Apr 2 12:47:30 turris foris-controller[11094]: DEBUG:foris_controller.buses.mqtt:Clearing retained messages 'foris-controller/0000000B00009CF7/reply/76662fce-95af-4cac-b582-e5d806c8e58a' Apr 2 12:47:30 turris mosquitto[12446]: 1585831650: New connection from ::1 on port 11883. Apr 2 12:47:30 turris mosquitto[12446]: 1585831650: New client connected from ::1 as auto-7B2424F3-FF47-69DC-A065-2DB373897A87 (p2, c1, k60, u'local'). Apr 2 12:47:30 turris mosquitto[12446]: 1585831650: No will message specified. Apr 2 12:47:30 turris mosquitto[12446]: 1585831650: Sending CONNACK to auto-7B2424F3-FF47-69DC-A065-2DB373897A87 (0, 0) Apr 2 12:47:30 turris mosquitto[12446]: 1585831650: Received PUBLISH from auto-7B2424F3-FF47-69DC-A065-2DB373897A87 (d0, q2, r1, m1, 'foris-controller/0000000B00009CF7/reply/76662fce-95af-4cac-b582-e5d806c8e58a', ... (0 bytes))@mmatejek does it ring the bell? :)
Bogdan and Martin found out, that setting static IP to MOX does not work.
Edited by Michal HruseckyRelated to it, could you explain me why the
MOXgets two IP addresses? (It is connected via LAN cable toOMNIA.)Hostname IPv4-Address MAC-Address Leasetime remaining 0000000D30007A23 192.168.1.203 D8:58:xx:xx:xx:56 7h 26m 54s turris 192.168.1.123 D8:58:xx:xx:xx:57 7h 29m 12s And why it is called "turris", when I renamed the device (in the FORIS tab) to "mox"? Thanks
Edited by Robert Ivánek-
Most likely it's a bug, see https://gitlab.labs.nic.cz/turris/foris/foris-subordinates-plugin/issues/6#note_148823
It is possible that the router (omnia) is connecting to a wrong IP address. Unfortunately auto discovery hasn't been implemented yet and router is trying to access IP which does not belong to the mox -> it fails.
Can you please check that IP address from your
/etc/config/fosquittoon the router matches the current IP of your mox device?Related to it, could you explain me why the MOX gets two IP addresses?
It seems a bit strange. Because there are two different mac addresses. Does your netbooted mox have two eth ports?
Anyways it might be a good idea to check
/etc/config/dhcpand see which mac is assigned to which IP.Note that the required state is that MAC address of your mox device is assigned to IP address (
/etc/config/dhcp) which is used in/etc/config/fosquitto.There might've been some flaw in the registration procedure of your netbooted mox.
-
It seems a bit strange. Because there are two different mac addresses. Does your netbooted mox have two eth ports?
For some reason, at first connection MAC address of wan device is used and then when bridge is configured, it gets MAC address of wan + 1.
E.g.
wan aa:bb:cc:dd:ee:74 br-lan aa:bb:cc:dd:ee:75At least that is what I see on my mox.
I have checked
/etc/config/fosquittoconfig global 'global' option debug '0' config local 'local' option port '11883' config remote 'remote' option enabled '0' config subordinate '0000000D30007A23' option address '192.168.1.203' option port '11884' option enabled '1'and
/etc/config/dhcpconfig host '0000000D30007A23' option mac 'd8:58:xx:xx:xx:56' option name '0000000D30007A23' option ip '192.168.1.203'@mmatejek, I do not understand, could you please explain in more details? why there are two addresses? And is that correct (expected) behavior?
Edited by Robert IvánekI revoke the pairing and did it again. I ended up in the same situation. Again two addresses
.123and.203/etc/config/fosquittostayed the same but/etc/config/dhcpchangedconfig host '0000000D30007A23' option mac 'd8:58:xx:xx:xx:56' option name '0000000D30007A23' option ip '192.168.1.202'So now they even do not match each other...
Edited by Robert Ivánek-
What mox model do you have? This didn't manifest on Mox with just wifi.
On the other hand, if you also have ethernet module, than I can reproduce it here.
Mox gets IP address twice during netboot process.
At first mox gets IP address through
eth0interface with MAC address (let's sayD8:58:xx:xx:xx:56) and send request for pairing or authenticate itself. After that it gets rootfs from omnia. Then it reshuffle network interfaces and request new IP address from interfacebr-lan- bridge that includeseth0- with MAC address of that bridge, thus mox might get additional IP address.And there is the culprit. In case of "just wifi" mox, MAC address of that bridge is the same as of
eth0and it works fine.In case of additional ethernet module, bridge MAC address is different (let's say
D8:58:xx:xx:xx:57) and thus mox will get second IP address. You "hit the nail on the head".
I have Mox Classic, with G module I believe. Are you planning to fix it also for Mox with ethernet module?
Or the entire netboot was only meant for Mox with just wifi
Edited by Robert IvánekI did that, however now single IP is assigned to two MAC addresses:
Expires IP Address MAC Address Hostname 2020-04-04 08:20 192.168.1.202 D8:58:xx:xx:xx:57 0000000D30007A23 2020-04-04 08:18 192.168.1.202 D8:58:xx:xx:xx:56 0000000D30007A23 and the
pingdoes work anymore:$ ping -c 5 192.168.1.202 PING 192.168.1.202 (192.168.1.202) 56(84) bytes of data. From 192.168.1.1 icmp_seq=1 Destination Host Unreachable From 192.168.1.1 icmp_seq=2 Destination Host Unreachable From 192.168.1.1 icmp_seq=3 Destination Host Unreachable From 192.168.1.1 icmp_seq=4 Destination Host Unreachable From 192.168.1.1 icmp_seq=5 Destination Host UnreachableIn the original stage (addresses 192.168.1.123 and 192.168.1.203)
pingonly worked on 192.168.1.123. 192.168.1.203 was also unreachable.Edited by Robert IvánekEDIT!
After I rebooted also OMNIA. I got to the functional stage:
- IP addresses:
Expires IP Address MAC Address Hostname 2020-04-04 09:21 192.168.1.202 D8:58:xx:xx:xx:57 0000000D30007A23 2020-04-04 09:19 192.168.1.122 D8:58:xx:xx:xx:56 * -
I can
ping192.168.1.202 and not the 192.168.1.122. -
The sub-tabs in Foris are functional, both
- set-up
- wifi
Wow, not bad for Friday evening! And thanks!
Edited by Robert IvánekEDIT 2: I observed that after some time (~12hours) the Mox gets in weird state. The LED changes from heartbeat to always-on mode. And there are suddenly no channels (there is empty field) nor devices (there is 0) connected. Despite the fact that there are devices in the neighborhood, which would be normally connected to it. After running
netboot-manager-hbk regen-fthe Mox gets back to operational state (LED heartbeat, channels and devices again present).P.S. I was asking in the forum how to turn-off the LED but I am not trying it yet. First I simply want to see, that Mox reliably works.
I switched on MOX back to
HBSbranch and it seems to be more stable. It is still running after 12 hours.Edited by Robert Ivánek-
Thanks for the observations.
However this is separate issue and we should discuss it elsewhere. If you find additional details, please update this issue.
mentioned in issue turris/turris-netboot#4 (closed)
added Doing label
Hi,
The Mox (still using
hbs) is again in the weird state after upgrading Turris OS on Omnia. (In Foris, subtab shows for device exclamation mark, no channels in wifi subtab, only rolling circles).neboot-manager regen -fdid not help. The IP is set as you recommended last time.OMNIA: Turris OS Version: 5.0.0 Release Branch: hbt Release Revision: a8c92e9 Kernel: 4.14.179Could you also suggest how to debug? In turn on debugging in
/etc/config/fosquittoand now see these messages in log:May 12 14:16:26 turris mosquitto[1394]: 1589292986: Bridge local.turris.0000000D30007A23 doing local SUBSCRIBE on topic foris-controller/0000000D30007A23/request/+/action/+ May 12 14:16:26 turris mosquitto[1394]: 1589292986: Bridge local.turris.0000000D30007A23 doing local SUBSCRIBE on topic foris-controller/0000000D30007A23/list May 12 14:16:26 turris mosquitto[1394]: 1589292986: Bridge local.turris.0000000D30007A23 doing local SUBSCRIBE on topic foris-controller/0000000D30007A23/request/+/list May 12 14:16:26 turris mosquitto[1394]: 1589292986: Bridge local.turris.0000000D30007A23 doing local SUBSCRIBE on topic foris-controller/0000000D30007A23/list May 12 14:16:26 turris mosquitto[1394]: 1589292986: Bridge local.turris.0000000D30007A23 doing local SUBSCRIBE on topic foris-controller/0000000D30007A23/schema May 12 14:16:26 turris mosquitto[1394]: 1589292986: Connecting bridge 0000000D30007A23 (192.168.1.202:11884) May 12 14:16:26 turris mosquitto[1394]: 1589292986: Bridge turris.0000000D30007A23 sending CONNECT May 12 14:16:29 turris mosquitto[1394]: 1589292989: Socket error on client local.turris.0000000D30007A23, disconnecting.Not so sure if these errors are also relevant:
May 12 19:09:37 turris mosquitto[4977]: 1589310577: OpenSSL Error[0]: error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding May 12 19:09:37 turris mosquitto[4977]: 1589310577: OpenSSL Error[1]: error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed May 12 19:09:37 turris mosquitto[4977]: 1589310577: OpenSSL Error[2]: error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib May 12 19:09:37 turris mosquitto[4977]: 1589310577: OpenSSL Error[3]: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failedEdited by Robert IvánekI managed to get it connected and responsive again.
These steps were done (not sure if all reboots were required):
- revoking the device (
netboot-manager revoke <id>) - deleting corresponding section in
/etc/config/fosquitto - deleting corresponding section in
/etc/config/dhcp - reboot Omnia
- factory reset Mox
- accepting the device (
netboot-manager accept <id>) - changing the MAC address in
/etc/config/dhcpto the second MAC (as I have Mox Classic with additional ethernet ports) - reboot Omnia
-> Finally it works again. No exclamation state. It shows OK in
ForisandreForis.- revoking the device (
-
@mhrusecky and @mmatejek is there any progress?
-
I see two things that we should address here:
- MAC address handling on netbooted mox to avoid this "disconnected state" in first place
- make sure that already established connection stay connected and won't break in certain cases
Unfortunately I'm not that familiar with any of these parts, therefore I don't think I'm the right person to handle that.
Hi let me clarify on that a bit. The problem is following:
- The mox (net)boots for the first time. It has some limited image which uses only WAN eth port with macaddr of the port.
- User registers it on omnia. At this point a dhcp record is created on omnia containing the macaddr of the port and a new static IP address.
- Mox obtains a new image and fully boots (or may be rebooted). When it has an extra port it automatically creates a bridge (e.g.
br-lan) with another mac address. And omnia assignes a dynamic IP to the mox. But records in/etc/config/fosquittoare pointing to the static address, which is not reachable (this will cause theexclamation mark).
From my point of view there are three possible solution for such problem.
- Figure out the mac address which would be assigned to
br-lanand assign both macs for the same IP address. (This could be a bit hacky because thebr-lanto mac assignment is a kind of openwrt blackbox) - User can access the omnia and manually update the new mac address in the LAN tab. Not sure whether it is implemented in (re)foris though. But there is an api call to create static leases (see https://gitlab.labs.nic.cz/turris/foris-controller/foris-controller/-/blob/master/foris_controller_modules/lan/schema/lan.json#L185 )
- implement automatic discovery for subordinate devices (this seems to be a lot of work, but probably is the right approach)
Note that currently I added an api call so it will be possible to change subordinates IP address. It could help here as well.
changed milestone to %Turris OS 5.0.1
This waits for @mhrusecky. It is clear that we can't count on exact release. It is rather when he has time so I am removing milestone for now.
Hi, It happened again with update of Omnia to
5.0.1, the Mox got non-responsive again. There are only spinning wheels in the wifi subtab.So I have to go through the revoking/rebooting/accepting procedure again. This unfortunately makes the
netbootfeature not so "carefree" :(Can I help with some debugging/logs?
Edited by Robert Ivánekmarked this issue as related to turris/turris-netboot#4 (closed)
This issue has no activity for more than 3 years. It is probably currently irrelevant (already fixed, etc.) and should be closed. I will close it now. @mhrusecky, if you consider it still relevant, please reopen it.
removed Doing label
