deckard issueshttps://gitlab.nic.cz/knot/deckard/-/issues2022-08-29T16:32:09+02:00https://gitlab.nic.cz/knot/deckard/-/issues/74Add missing dependecy to readme.md2022-08-29T16:32:09+02:00AureliaAdd missing dependecy to readme.mdOn Ubuntu 22.04.1 LTS I needed to install Ubuntu [dpkt](https://dpkt.readthedocs.io/en/latest/)
I can open the related MR and update the readme.On Ubuntu 22.04.1 LTS I needed to install Ubuntu [dpkt](https://dpkt.readthedocs.io/en/latest/)
I can open the related MR and update the readme.https://gitlab.nic.cz/knot/deckard/-/issues/73Update Readme.md with faketime link2022-08-29T15:56:29+02:00AureliaUpdate Readme.md with faketime linkIn the readme, [here](https://gitlab.nic.cz/knot/deckard/-/blob/master/README.rst?plain=1#L37), the actual link to faketime is missing at the end of the file.In the readme, [here](https://gitlab.nic.cz/knot/deckard/-/blob/master/README.rst?plain=1#L37), the actual link to faketime is missing at the end of the file.https://gitlab.nic.cz/knot/deckard/-/issues/72bind9 - many test cases end with 'unsupported as of yet'2022-03-24T11:11:45+01:00perlangbind9 - many test cases end with 'unsupported as of yet'I have tried it with the newest version BIND-9.18.0.
The BIND-9.18.0 is compiled with '--enable-full-report' '--enable-dnstap' '--without-jemalloc',
by gcc 8.5.0.
I get the followed summary result,
`====== 251 failed, 30 passed, 99 sk...I have tried it with the newest version BIND-9.18.0.
The BIND-9.18.0 is compiled with '--enable-full-report' '--enable-dnstap' '--without-jemalloc',
by gcc 8.5.0.
I get the followed summary result,
`====== 251 failed, 30 passed, 99 skipped, 1 warning in 377.15s (0:06:17) =======`
about 171 failed cases have the wrong syntax configure file named.conf which
include a line like "unsupported as of yet" on line 20.
my environment is RockyLinux 8.5,
the libfaketime is from epel, others is from rocky's repo,
and the python modules are installed via
`# pip3 install -r requirements`
python version is 3.9.6 (from rocky).https://gitlab.nic.cz/knot/deckard/-/issues/70trouble on centos72022-03-24T11:13:35+01:00wzlutrouble on centos7I try to run deckard on centos7:
CentOS Linux release 7.9.2009 (Core)
uname -a: Linux localhost.localdomain 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Python 3.7.11
But receive the foll...I try to run deckard on centos7:
CentOS Linux release 7.9.2009 (Core)
uname -a: Linux localhost.localdomain 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Python 3.7.11
But receive the following messages:
(base) [lwz@localhost deckard-master]$ ./kresd_run.sh -n 4 -k "iter_minim_a.rpl"
[gw0] Python 3.7.11 (default, Jul 27 2021, 14:32:16) -- [GCC 7.5.0]
[gw1] Python 3.7.11 (default, Jul 27 2021, 14:32:16) -- [GCC 7.5.0]
[gw2] Python 3.7.11 (default, Jul 27 2021, 14:32:16) -- [GCC 7.5.0]
[gw3] Python 3.7.11 (default, Jul 27 2021, 14:32:16) -- [GCC 7.5.0]
gw0 ok / gw1 ok / gw2 ok / gw3 ok
gw0 [0] / gw1 [0] / gw2 [0] / gw3 [0]
scheduling tests via LoadScheduling
============================================================ warnings summary =============================================================
../../ana3/lib/python3.7/site-packages/xdist/plugin.py:205
../../ana3/lib/python3.7/site-packages/xdist/plugin.py:205
../../ana3/lib/python3.7/site-packages/xdist/plugin.py:205
../../ana3/lib/python3.7/site-packages/xdist/plugin.py:205
../../ana3/lib/python3.7/site-packages/xdist/plugin.py:205
/home/lwz/ana3/lib/python3.7/site-packages/xdist/plugin.py:205: DeprecationWarning: The --boxed commmand line argument is deprecated. Install pytest-forked and use --forked instead. pytest-xdist 3.0.0 will remove the --boxed argument and pytest-forked dependency.
config.issue_config_time_warning(warning, 2)
../../ana3/lib/python3.7/site-packages/pyroute2/__init__.py:72
../../ana3/lib/python3.7/site-packages/pyroute2/__init__.py:72
../../ana3/lib/python3.7/site-packages/pyroute2/__init__.py:72
../../ana3/lib/python3.7/site-packages/pyroute2/__init__.py:72
/home/lwz/ana3/lib/python3.7/site-packages/pyroute2/__init__.py:72: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
for entry_point in metadata.entry_points().get('pr2modules', []):
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================================================== 9 warnings in 2.48s ===========================================================
[gw0] Python 3.7.11 (default, Jul 27 2021, 14:32:16) -- [GCC 7.5.0]
[gw1] Python 3.7.11 (default, Jul 27 2021, 14:32:16) -- [GCC 7.5.0]
[gw2] Python 3.7.11 (default, Jul 27 2021, 14:32:16) -- [GCC 7.5.0]
[gw3] Python 3.7.11 (default, Jul 27 2021, 14:32:16) -- [GCC 7.5.0]
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2]
scheduling tests via LoadScheduling
deckard_pytest.py::test_passes_qmin_on[Scenario(path='sets/resolver/iter_minim_a.rpl', qmin=True, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3]
deckard_pytest.py::test_passes_qmin_off[Scenario(path='sets/resolver/iter_minim_a.rpl', qmin=True, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3]
[gw0] [ 50%] ERROR deckard_pytest.py::test_passes_qmin_on[Scenario(path='sets/resolver/iter_minim_a.rpl', qmin=True, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3]
[gw1] [100%] ERROR deckard_pytest.py::test_passes_qmin_off[Scenario(path='sets/resolver/iter_minim_a.rpl', qmin=True, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3]
================================================================= ERRORS ==================================================================
_ ERROR at setup of test_passes_qmin_on[Scenario(path='sets/resolver/iter_minim_a.rpl', qmin=True, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3] _
[gw0] linux -- Python 3.7.11 /home/lwz/ana3/bin/python3
conftest.py:126: in pytest_runtest_setup
LinuxNamespace("user").__enter__()
contrib/namespaces.py:60: in __init__
raise OSError(e, os.strerror(e))
E OSError: [Errno 22] Invalid argument
_ ERROR at setup of test_passes_qmin_off[Scenario(path='sets/resolver/iter_minim_a.rpl', qmin=True, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3] _
[gw1] linux -- Python 3.7.11 /home/lwz/ana3/bin/python3
conftest.py:126: in pytest_runtest_setup
LinuxNamespace("user").__enter__()
contrib/namespaces.py:60: in __init__
raise OSError(e, os.strerror(e))
E OSError: [Errno 22] Invalid argument
============================================================ warnings summary =============================================================
../../ana3/lib/python3.7/site-packages/xdist/plugin.py:205
../../ana3/lib/python3.7/site-packages/xdist/plugin.py:205
../../ana3/lib/python3.7/site-packages/xdist/plugin.py:205
../../ana3/lib/python3.7/site-packages/xdist/plugin.py:205
../../ana3/lib/python3.7/site-packages/xdist/plugin.py:205
/home/lwz/ana3/lib/python3.7/site-packages/xdist/plugin.py:205: DeprecationWarning: The --boxed commmand line argument is deprecated. Install pytest-forked and use --forked instead. pytest-xdist 3.0.0 will remove the --boxed argument and pytest-forked dependency.
config.issue_config_time_warning(warning, 2)
../../ana3/lib/python3.7/site-packages/pyroute2/__init__.py:72
../../ana3/lib/python3.7/site-packages/pyroute2/__init__.py:72
../../ana3/lib/python3.7/site-packages/pyroute2/__init__.py:72
../../ana3/lib/python3.7/site-packages/pyroute2/__init__.py:72
/home/lwz/ana3/lib/python3.7/site-packages/pyroute2/__init__.py:72: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
for entry_point in metadata.entry_points().get('pr2modules', []):
-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================= short test summary info =========================================================
ERROR deckard_pytest.py::test_passes_qmin_on[Scenario(path='sets/resolver/iter_minim_a.rpl', qmin=True, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3]
ERROR deckard_pytest.py::test_passes_qmin_off[Scenario(path='sets/resolver/iter_minim_a.rpl', qmin=True, config={'programs': [{'name': 'kresd', 'binary': 'kresd', 'additional': ['-n'], 'templates': ['template/kresd.j2'], 'configs': ['config']}]})-max-retries-3]
====================================================== 9 warnings, 2 errors in 2.45s ======================================================https://gitlab.nic.cz/knot/deckard/-/issues/69Introduce a new format to describe (not only) Deckard scenarios2021-03-18T12:06:32+01:00Štěpán BalážikIntroduce a new format to describe (not only) Deckard scenariosI'll summarize reasons, caveats and discussion points of this in separate threads in this issue.I'll summarize reasons, caveats and discussion points of this in separate threads in this issue.Štěpán BalážikŠtěpán Balážikhttps://gitlab.nic.cz/knot/deckard/-/issues/68investigate random fails in knot-resolver CI2020-12-18T14:08:47+01:00Štěpán Balážikinvestigate random fails in knot-resolver CIThe `deckard` job often fails in the CI [like this](https://gitlab.nic.cz/knot/knot-resolver/-/jobs/472294) with `E RuntimeError: Server took too long to respond` and other random fail which in the current setup will trigger a rerun of...The `deckard` job often fails in the CI [like this](https://gitlab.nic.cz/knot/knot-resolver/-/jobs/472294) with `E RuntimeError: Server took too long to respond` and other random fail which in the current setup will trigger a rerun of the whole test suite.
Investigate the cause of the random errors, if that becomes too hard, consider adding some logic to rerun only the randomly failed tests (there is some retry logic in Deckard already, so maybe use that). Alternatively, making the timeout for queries longer might help as well.https://gitlab.nic.cz/knot/deckard/-/issues/67extension for NSEC(3) range handling in scenarios2022-02-07T18:19:54+01:00Petr Špačekextension for NSEC(3) range handling in scenariosCurrently RPL format allows only exact matches on qname and/or qtype or subdomain match. This leads to significant duplication of answers with NSEC/NSEC3 replies because we have to define several exact match entries which ultimately prov...Currently RPL format allows only exact matches on qname and/or qtype or subdomain match. This leads to significant duplication of answers with NSEC/NSEC3 replies because we have to define several exact match entries which ultimately provide the same set of NSEC/NSEC3 records.
Maybe we could implement something like `MATCH nsec` or `MATCH nsec3` and put a NSEC(3) record into QUESTION section on the entry (or elsewhere). Then Deckard would use the entry only if it belongs to the given range.https://gitlab.nic.cz/knot/deckard/-/issues/66testserver does not respect EDNS buffer size2020-10-21T21:29:06+02:00Petr Špačektestserver does not respect EDNS buffer sizeTestserver currently hardcodes EDNS version 0 buffer size 4096 and ignores EDNS parameters sent by client. This leads to situation where testserver might send back oversized UDP response which is later refused by client.
We need to disc...Testserver currently hardcodes EDNS version 0 buffer size 4096 and ignores EDNS parameters sent by client. This leads to situation where testserver might send back oversized UDP response which is later refused by client.
We need to discuss what to do with that. Current behavior breaks tests if resolver under test is using different value than 4096 B.
Options:
a) Add ability to specify alternative ENTRY when first matching ENTRY does not fit into EDNS buffer. This is probably too verbose because in vast majority of cases we want to send back empty response with TC bit and let client fall back to TCP.
b) Detect that current message size if bigger than min(client, Deckard) values and automatically generate response with TC=1. Do we need ability to override this automatic behavior?
TODO: Check if queries over TCP actually work with answers bigger than EDNS buffer size.https://gitlab.nic.cz/knot/deckard/-/issues/65all recently imported tests from testbound might be broken (i.e. not test any...2021-01-25T20:06:33+01:00Štěpán Balážikall recently imported tests from testbound might be broken (i.e. not test anything)The following discussion from !153 should be addressed:
> Procedure for generating seems flawed:
>
> Upstream [black_prime.rpl](https://github.com/NLnetLabs/unbound/blob/68c57314c4e02079569ed697b9273fa9faaafeb6/testdata/black_prime.rpl...The following discussion from !153 should be addressed:
> Procedure for generating seems flawed:
>
> Upstream [black_prime.rpl](https://github.com/NLnetLabs/unbound/blob/68c57314c4e02079569ed697b9273fa9faaafeb6/testdata/black_prime.rpl) has intentionally expired RRSIGs for the whole `example.com` in `RANGE` with `ADDRESS 1.2.3.4` but the generator puts valid signatures in the resulting [file](https://gitlab.nic.cz/knot/deckard/-/blob/98bd7a71f24e76cc278593a42bfe07b3fac7b6d5/sets/resolver/black_prime.rpl).
>
> This means that any of the tests merged in !73, !141, !146 and !148 might actually not test anything.
>
> I'll go through the `black_*` tests manually as there the mistakes are obvious; others will have to wait for a fix for the generator.https://gitlab.nic.cz/knot/deckard/-/issues/64ci: XML reports used in comparative tests are broken2021-04-30T16:18:46+02:00Štěpán Balážikci: XML reports used in comparative tests are brokenThis job for example https://gitlab.nic.cz/knot/deckard/-/jobs/438159 should not pass since some test fail in the first run and pass in the second one without any modification.
This is caused by the `head.xml` and `base.xml` reports fro...This job for example https://gitlab.nic.cz/knot/deckard/-/jobs/438159 should not pass since some test fail in the first run and pass in the second one without any modification.
This is caused by the `head.xml` and `base.xml` reports from pytest being generated almost empty for some reason.
I suppose this broke in the _recent_ pytest update.https://gitlab.nic.cz/knot/deckard/-/issues/63Reintroduce a way to access other binaries' IP addresses (and maybe all their...2020-11-26T12:23:10+01:00Štěpán BalážikReintroduce a way to access other binaries' IP addresses (and maybe all their configuration) from Jinja templateBefore 04d2e67355ce5e3d34844a808f0701df72aea07f there was a IPADDRS dictionary indexed with a binary's name for this purpose but it got scrapped in the networking rewrite (but not [from](https://gitlab.nic.cz/knot/deckard/-/blob/06bfe7fa...Before 04d2e67355ce5e3d34844a808f0701df72aea07f there was a IPADDRS dictionary indexed with a binary's name for this purpose but it got scrapped in the networking rewrite (but not [from](https://gitlab.nic.cz/knot/deckard/-/blob/06bfe7fa097a42d507acdc1399436246a9ed7edc/doc/user_guide.rst#L121) [the](https://gitlab.nic.cz/knot/deckard/-/blob/06bfe7fa097a42d507acdc1399436246a9ed7edc/doc/user_guide.rst#206) [docs](https://gitlab.nic.cz/knot/deckard/-/blob/06bfe7fa097a42d507acdc1399436246a9ed7edc/doc/user_guide.rst#L104)).
Apparently usecases for this exist (testing forwarding is certainly one of them), so let's reintroduce some a more general of accessing other programs' configuration: I suggest a `PROGRAMS` dictionary indexed (again) by name, which exposes all the configuration fields. One would than use `PROGRAMS["recursor"]["address"]` instead of `IPADDRS["recursor"]`. We might document as much and as few of them that we dare to pronounce stable enough.Štěpán BalážikŠtěpán Balážikhttps://gitlab.nic.cz/knot/deckard/-/issues/62a2020-08-07T16:06:13+02:00Petr Špačekaknot/knot-resolver/!1020knot/knot-resolver/!1020https://gitlab.nic.cz/knot/deckard/-/issues/61a2020-08-07T16:06:07+02:00Petr Špačekaknot/knot-resolver/!1020knot/knot-resolver/!1020https://gitlab.nic.cz/knot/deckard/-/issues/60a2020-08-07T16:06:02+02:00Petr Špačekaknot/knot-resolver/!1020knot/knot-resolver/!1020https://gitlab.nic.cz/knot/deckard/-/issues/59a2020-08-07T16:05:58+02:00Petr Špačekaknot/knot-resolver/!1020knot/knot-resolver/!1020https://gitlab.nic.cz/knot/deckard/-/issues/58a2020-08-07T16:05:53+02:00Petr Špačekaknot/knot-resolver/!1020knot/knot-resolver/!1020https://gitlab.nic.cz/knot/deckard/-/issues/57find smarter way of describing expected DNS answers2021-02-02T11:10:21+01:00Petr Špačekfind smarter way of describing expected DNS answersProblem
=======
Currently `CHECK_ANSWER` step takes DNS answer message and matches specified fields against a hardcoded DNS message in RPL file. This is especially problem for answers in border cases where exact content is not 100 % spec...Problem
=======
Currently `CHECK_ANSWER` step takes DNS answer message and matches specified fields against a hardcoded DNS message in RPL file. This is especially problem for answers in border cases where exact content is not 100 % specified in the DNS protocol or depends on content of DNS cache etc. E.g. server might add just subset of glue addresses into additional section, or it might add NS records to authority section (i.e. non-minimal answers) etc. This is impossible to represent in the current RPL format.
Idea
====
Mark RRs in `RANGE` sections with "permissibility" for different sections in answers. E.g.:
- bogus RR would be marked "no section unless CD bit is on"
- NS records would be marked "optional in authority section unless QTYPE=NS"
etc.
This should lead to tests which are not too sensitive to random RR selection or minor changes in selection on the responder side.https://gitlab.nic.cz/knot/deckard/-/issues/56randomize order of steps in idempotent tests2020-08-07T11:07:59+02:00Petr Špačekrandomize order of steps in idempotent testsProblem
=======
Current RPL tests are quite static and often do not test various combinations of cache state.
Idea
====
Tests which use one static set of answers (i.e. all QUERY/CHECK_ANSWER steps belong to single numeric range which co...Problem
=======
Current RPL tests are quite static and often do not test various combinations of cache state.
Idea
====
Tests which use one static set of answers (i.e. all QUERY/CHECK_ANSWER steps belong to single numeric range which covers all RANGE_BEGIN entries) should produce the same DNS answers even if QUERY steps are run in different order.
For these "idempotent" tests Deckard could:
- [ ] randomize ordering of QUERY+CHECK pairs
- [ ] randomize delay between QUERY steps (based on TTLs in the test?) or maybe in combination with randomizing TTLs
- [ ] continue testing after a failed step? run the scenario again and skip the failed test?
- [ ] run steps in isolation?https://gitlab.nic.cz/knot/deckard/-/issues/55investigate warning: unknown key log_print in deckard_pytest.ini2020-08-06T13:09:59+02:00Petr Špačekinvestigate warning: unknown key log_print in deckard_pytest.iniCommit 5708002a29f3824c618fde02224edba89aa63c2f by @sbalazik introduced key `log_print` into `deckard_pytest.ini` but apparently it is not implemented in py.test 6.0.0.
Please investigate if the key is needed or if it can be safely remo...Commit 5708002a29f3824c618fde02224edba89aa63c2f by @sbalazik introduced key `log_print` into `deckard_pytest.ini` but apparently it is not implemented in py.test 6.0.0.
Please investigate if the key is needed or if it can be safely removed.Štěpán BalážikŠtěpán Balážikhttps://gitlab.nic.cz/knot/deckard/-/issues/54resurrect RFC 5011 tests2020-08-12T11:05:32+02:00Petr Špačekresurrect RFC 5011 testsThese test were removed in https://gitlab.nic.cz/knot/deckard/-/commit/192c269164a9e29c3199c679a57a50cb96806eba but we need them back, RFC 5011 in Knot Resolver got broken without us noticing.
Problems:
- [ ] might require faking monoto...These test were removed in https://gitlab.nic.cz/knot/deckard/-/commit/192c269164a9e29c3199c679a57a50cb96806eba but we need them back, RFC 5011 in Knot Resolver got broken without us noticing.
Problems:
- [ ] might require faking monotonic time as well as real time clock
- [ ] probably requires different kresd config template to enable `ta_update` module -> move test to kresd repo?Štěpán BalážikŠtěpán Balážik