File 6291-ct-add-internal_doc-and-ct_notes.md.patch of Package erlang

Overview Repositories Revisions Requests Users Attributes Meta

File 6291-ct-add-internal_doc-and-ct_notes.md.patch of Package erlang (Revision 3e53aa07e56d6cc0c5ffbbfd2f14b043)

Currently displaying revision 3e53aa07e56d6cc0c5ffbbfd2f14b043 , Show latest

From 87291144f9c78e9390c5fdf2905454829368276c Mon Sep 17 00:00:00 2001
From: Jakub Witczak <kuba@erlang.org>
Date: Tue, 20 Jun 2023 18:32:11 +0200
Subject: [PATCH] ct: add internal_doc and ct_notes.md

---
 lib/common_test/internal_doc/ct_notes.md | 161 +++++++++++++++++++++++
 1 file changed, 161 insertions(+)
 create mode 100644 lib/common_test/internal_doc/ct_notes.md

diff --git a/lib/common_test/internal_doc/ct_notes.md b/lib/common_test/internal_doc/ct_notes.md
new file mode 100644
index 0000000000..84c70b97ef
--- /dev/null
+++ b/lib/common_test/internal_doc/ct_notes.md
@@ -0,0 +1,161 @@
+# CT test_server
+
+## Problem (GH-7119, OTP-11894, OTP-14480)
+I think the most confusing thing is that today OTP behavior and design seems to be a mix of Configuration and Testcase centric attributes:
+1. (Configuration centric) CT hook callback looks as designed to wrap around CT Configuration functions (i.e. you have *pre* and *post* to wrapp around init_per_testcase or end_per_testcase)
+   - Furthermore if you consider hook callback function names, there are no hooks wrapping around Testcase function at all!
+2. (Testcase centric) AND at the same the hook execution order is determined by relation to CT Testcase callback
+### Next step ideas
+1. improve existing documentation for hooks (actually it was planned for many years but down prioritized)
+2. add mermaid diagrams to docs when that is possible
+3. introduce a CT option for Configuration centric hook execution order (maybe named ct_hooks_order := [testcase(default) | configuration])
+### CT hooks priorities (documentation sketch)
+Let's assume:
+1. cth_A and cth_B being CT hook modules to be installed
+2. A, B are CT hook priorities where A is more important than B (higher priority)
+3. hook priorities affect order in which hook callbacks of the same type are executed
+4. Testcase functions - are CT callback functions defined in suite which will have test verdict assigned (e.g. ok, fail, skip)
+5. Configuration functions - are CT callback functions defining initialization and cleanup routines for testcases (e.g. init_per_testcase, end_per_testcase)
+
+```mermaid
+---
+title: Diagram legend
+---
+flowchart LR
+    dc((("double circle"))) -.- i>item determining hook execution order]
+    pgr[/"parallelogram"/] -.- j>item not relevant for hook execution order]
+```
+
+#### Testcase centric (default)
+1. execution order is related to position in relation to Testcase
+2. all *init* hook callbacks(positioned before Testcase) are executed in some order 
+3. all *end* hook callbacks(positioned after Testcase) are executed in reversed order
+4. *pre* and *post* hook callback type is not affecting execution order
+```mermaid
+---
+title: Testcase centric CT hook execution order (default)
+---
+flowchart TD
+    subgraph hooks
+    pre_init_pt_A["(A) pre_init_per_testcase"] --> pre_init_pt_B
+    end
+    subgraph suite
+    pre_init_pt_B["(B) pre_init_per_testcase"] --> init_pt[/"init_per_testcase"/]
+    end
+    init_pt --> post_init_pt_A
+    subgraph hooks
+    post_init_pt_A["(A) post_init_per_testcase"] --> post_init_pt_B
+    end
+    subgraph suite
+    post_init_pt_B["(B) post_init_per_testcase"] --> testcase
+    testcase((("Testcase")))
+    end
+    subgraph hooks
+    testcase --> pre_end_pt_B
+    pre_end_pt_B["(B) pre_end_per_testcase"] --> pre_end_pt_A
+    end
+    subgraph suite
+    pre_end_pt_A["(A) pre_end_per_testcase"] --> end_per_test_case
+    end
+    subgraph hooks
+    end_per_test_case[/"end_per_testcase"/] --> post_end_pt_B
+    post_end_pt_B["(B) post_end_per_testcase"] --> post_end_pt_A["(A) post_end_per_testcase"]
+    end
+```
+#### Configuration centric (option candidate)
+- all *pre* hooks will be executed in some order
+- all *post* hooks will be executed in reversed order
+- relation to Testcase is not relevant for hook execution order
+- Note: I think, it might be considered a bit inconsistent because there are no hook callbacks wrapping around Testcase (e.g. pre_testcase, post_testcase)
+- Above could be ugly workarounded by using pre_init_per_testcase with post_end_per_testcase
+- Adding pre_testcase, post_testcase might not be simple but maybe it is not needed
+
+```mermaid
+---
+title: Configuration centric CT hook execution order (option)
+---
+flowchart TD
+    subgraph hooks
+    pre_init_pt_A["(A) pre_init_per_testcase"] --> pre_init_pt_B
+    end
+    subgraph suite
+    pre_init_pt_B["(B) pre_init_per_testcase"] --> init_pt((("init_per_testcase")))
+    end
+    init_pt --> post_init_pt_B
+    subgraph hooks
+    post_init_pt_B["(B) post_init_per_testcase"] --> post_init_pt_A
+    end
+    subgraph suite
+    post_init_pt_A["(A) post_init_per_testcase"] --> testcase
+    testcase[/"Testcase"/]
+    end
+    subgraph hooks
+    testcase --> pre_end_pt_A
+    pre_end_pt_A["(A) pre_end_per_testcase"] --> pre_end_pt_B
+    end
+    subgraph suite
+    pre_end_pt_B["(B) pre_end_per_testcase"] --> end_per_test_case
+    end
+    subgraph hooks
+    end_per_test_case((("end_per_testcase"))) --> post_end_pt_B
+    post_end_pt_B["(B) post_end_per_testcase"] --> post_end_pt_A["(A) post_end_per_testcase"]
+    end
+```
+
+## processes
+```mermaid
+flowchart LR
+    tc["`tc
+         every CT callback function`"]
+    testcase["`testcase
+               the user Testcase`"]
+    testcase --> tc
+```
+
+## test_server code
+
+```mermaid
+mindmap
+    code
+        run_test_case_apply/1
+            run_test_case_apply/6
+                ))run_test_case_eval_fun/9((
+                    run_test_case_eval/9
+                        do_init_tc_call/4
+                        run_test_case_eval1/6
+                        do_end_tc_call/4
+                ct_util mark_process/0
+                run_test_case_msgloop/1
+                    [receive]
+        run_test_case_eval1/6
+            ts_tc/3
+```
+## processes
+```mermaid
+flowchart LR
+    job["`(job process GL)
+        captures output`"]
+    case["`(case process)
+        runs test suite`"]
+
+    case --news--> job
+```
+## concept
+```mermaid
+mindmap
+  root)test_server(
+    interfaces
+        sup
+            get_loc/1
+            set_ct_state/1
+        ctrl
+            run_test_case_apply/1
+            init_target_info/0
+            init_memory_checker/0
+        suite
+    init
+        target_info
+        memory_checker
+    questions
+        why call_end_conf not called when init/end_per_testcase not defined?
+```
-- 
2.35.3

Places

File 6291-ct-add-internal_doc-and-ct_notes.md.patch of Package erlang (Revision 3e53aa07e56d6cc0c5ffbbfd2f14b043)

Places