Tests¶

Two kinds of tests:

• ST tests in simulation (cts test): tests written in ST, in a test application that only ever runs in simulation. For the logic of POUs.
• JSON test plans (cts verify --with-test): variable writes and reads against an application online, on a PLC or in simulation.

When the project already has its test application (cts test runs rather than refuses), a new POU and its test take four steps: write the POU's .st at the top of project-view/ (shared by every application), add its Check/CheckReal calls to PRG_Tests.st, cts import, cts test. The rest of this guide sets the test application up and explains the plans.

ST tests in simulation¶

The POUs in the POUs view are shared by every application of the project. A second device holds the tests:

PLC_Main                      the device of the real PLC
  Plc Logic/Application       the production application (active)
Test                          a second device, always in simulation
  Plc Logic/TestApp
    Library Manager           with CmpLog (the runtime's log) and Standard
    TestResult                GVL: the counts cts reads
    TestLog                   FUNCTION: writes one CTS| record to the log
    Check, CheckReal          FUNCTION: count and log one check
    PRG_Tests                 PROGRAM: the tests
    Task Configuration/TestTask   calls PRG_Tests
POUs                          the project root, shared: the code under test
  F_Clamp

In the sync folder the project root is the top of project-view/ (project-view/F_Clamp.st); the test application's objects are under project-view/Test/Plc Logic/TestApp/.

cts test makes TestApp the active application, downloads it to the simulation, starts it, waits until TestResult.Done is TRUE, reads the counts, reads the simulation's log from the run's TEST_START record on, logs TestApp out and makes the earlier application active again. A device that is not in simulation is refused unless --plc says the tests may go to that PLC: then TestApp is downloaded there (the deny list's download applies) and the tests run on the real runtime. --repeat N runs them N times, each run after the first started again by a warm reset when the deny list allows reset_plc, else by another download (restart= on the line); the line counts the failed runs (failed_runs=) and names each failing test with its run, which finds a test that fails now and then. The records reach the log a moment after TestResult.Done: each run waits up to 5 s for its own TEST_START and as many results as the counts; a run whose TEST_START does not come did not run (its counts are an earlier run's) and fails. Exit 0 when every test of every run passed.

The wait ends early when the application stops on an exception, leaves run after it ran, or stops answering the reads of TestResult: the line says which, how long after the start and how many tests had counted, and quotes the exception the runtime logged (runtime: #### Exception: ...). A test that never ends without an exception (an endless loop) holds its task until --wait runs out; the error then says when the counts last moved. A watchdog on the test task (Task Configuration, TestTask, Watchdog) turns such a loop into an exception, which ends the wait at once. Known runtime lines in the run's log come with a hint: (TLS contexts used up, an exception).

Every run leaves its log: the records the tests wrote and whatever the runtime logged meanwhile (an exception, a watchdog), in the runtime's log file format. It goes to --out (a file, or a folder for <application>.log), else to <sync folder>/.git/cts/test/<application>.log; the line names it as log=.

cts test                        # TestApp
cts test --app UnitTests --wait 60 --out results/

ok test passed=3 failed=0 application=TestApp device=Test ms=5210 log=C:\plant\.git\cts\test\TestApp.log
error test passed=2 failed=1 application=TestApp device=Test ms=5180 log=C:\plant\.git\cts\test\TestApp.log
  FAIL F_Clamp cuts to hi got=11.0 want=10.0

--json lists every test as tests (name, status PASS/FAIL, fields) and the run's log entries as log.

Set it up once from files, by copying the PLC's (the same device type is fine) in project-view/:

• the device's .cds-object.xml to Test/, its Name changed to Test and its SimulationMode to True;
• its Plc Logic/.cds-object.xml to Test/Plc Logic/;
• the application's .cds-object.xml and Library Manager.xml to Test/Plc Logic/TestApp/, the Name changed to TestApp;
• its Task Configuration/.cds-object.xml, and one task file as Task Configuration/TestTask.xml, its Name changed to TestTask and PRG_Tests the only POU in its PouList.

Add the GVL, the functions and the program below as .st files in TestApp/, run cts lib add "CmpLog, * (System)" --app TestApp (Standard, for CONCAT, came with the copied Library Manager), then one cts import creates it all, outer objects first, and writes the copies back with the GUIDs the IDE gave them. A Name that differs from its file or folder is refused before the IDE. Only the code a task calls is downloaded, so the tests reach the shared POUs through PRG_Tests.

The result GVL, its names fixed:

{attribute 'qualified_only'}
VAR_GLOBAL
    Done : BOOL;        // TRUE once every test has run
    Passed : UDINT;
    Failed : UDINT;
END_VAR

The log records, one per runtime log entry (cts plc-log --cts reads the same format; an empty result with no malformed records points to the raw log tail):

CTS|M|TEST_START|<application>|      before the first test: the run's log starts here
CTS|M|TEST_PASS|<test>|
CTS|M|TEST_FAIL|<test>|k=v;k=v       what the test saw, e.g. got=11.0;want=10.0

A record is cut at 127 characters, so keep test names short, and a value holds neither | nor ;.

FUNCTION TestLog : BOOL
VAR_INPUT
    code : STRING(20);
    tag : STRING(100);
    fields : STRING(100);
END_VAR

CmpLog.LogAdd2(CmpLog.LogConstants.STD_LOGGER, 0, CmpLog.LogClass.LOG_INFO, 0, 0,
    CONCAT(CONCAT(CONCAT('CTS|M|', code), '|'), CONCAT(CONCAT(tag, '|'), fields)));
TestLog := TRUE;

One check, and one for numbers that logs what it got. It compares within a millionth (of the value, or absolute below 1): a computed LREAL rarely equals its decimal literal bit for bit.

FUNCTION Check : BOOL
VAR_INPUT
    ok : BOOL;
    name : STRING(100);
    detail : STRING(100);
END_VAR

IF ok THEN
    TestResult.Passed := TestResult.Passed + 1;
    TestLog('TEST_PASS', name, '');
ELSE
    TestResult.Failed := TestResult.Failed + 1;
    TestLog('TEST_FAIL', name, detail);
END_IF
Check := ok;

FUNCTION CheckReal : BOOL
VAR_INPUT
    got : LREAL;
    want : LREAL;
    name : STRING(100);
END_VAR

CheckReal := Check(ABS(got - want) <= 1.0E-6 * MAX(1.0, ABS(want)), name,
    CONCAT(CONCAT('got=', LREAL_TO_STRING(got)), CONCAT(';want=', LREAL_TO_STRING(want))));

The tests, run once:

PROGRAM PRG_Tests

IF NOT TestResult.Done THEN
    TestLog('TEST_START', 'TestApp', '');
    CheckReal(F_Clamp(5, 0, 10), 5, 'F_Clamp keeps a value inside');
    CheckReal(F_Clamp(-1, 0, 10), 0, 'F_Clamp lifts to lo');
    CheckReal(F_Clamp(11, 0, 10), 10, 'F_Clamp cuts to hi');
    TestResult.Done := TRUE;
END_IF

A new shared POU and its tests: create it at the project root, add its checks to PRG_Tests, carry both into the IDE and run the tests.

# pou: the object kind; --kind: which POU
cts new pou F_Lerp --kind function --return-type REAL --parent . \
  --text "VAR_INPUT a, b, t : REAL; END_VAR F_Lerp := a + (b - a) * t;"
# PRG_Tests.st: CheckReal(F_Lerp(0, 10, 0.5), 5, 'F_Lerp halfway');
cts import
cts test

cts test runs what the IDE holds, saved or not: no cts save is needed between the import and the test, and the test saves nothing.

A test that needs several cycles (a timer, a state machine) keeps its own step counter in PRG_Tests and sets Done when the last step has checked; --wait (seconds, default 30) is how long cts waits for it.

On a PLC (--plc), switch off the test application's boot application (TestApp -> Properties -> Boot application -> Implicit boot application on download); cts test names it while it is on. Otherwise every download leaves a boot application, and a restart of the runtime runs the tests again, a test that never ends included, so the restart does not free the PLC. To recover from such a loop: stop the runtime, move the test application's boot files (<App>.app and <App>.crc in the runtime's PlcLogic/<App>/) aside, start it, then cts test --plc downloads afresh.

JSON test plans¶

cts verify --with-test runs JSON test plans against a CODESYS online application. The in-IDE agent runs them (test.run), one agent call per step: login, start, variable writes and reads. A wait step does not hold the IDE, which stays usable while a plan runs.

Location¶

Test plans live in the project sync folder:

<sync-folder>/
  .test/
    arithmetic.json
    counter.json
  project-view/
  .dump/

Run one file:

cts verify --with-test --test-file arithmetic.json

Run all plans from .test/ in sorted order:

cts verify --with-test

The agent method directly:

cts raw run_tests --folder <sync-folder> --file arithmetic.json --timeout 120

Plan Object¶

Every plan names the CODESYS application under test, and that application must be the active one in the IDE: a plan for another application is refused, not switched to.

{
  "name": "Arithmetic smoke test",
  "application": "CI_CD_Application",
  "ip": "192.0.2.10",
  "gateway": "Gateway-1",
  "start": false,
  "timeout": 30000,
  "continue_on_fail": false,
  "tests": [
    {
      "name": "add",
      "steps": [
        { "action": "wait", "ms": 100 }
      ]
    }
  ]
}

Before each plan, the agent logs in to the active application (cts connect: no online change, no download), checks it is the plan's application, resets and starts it as the plan says, then executes the tests in order.

Test Object¶

{
  "name": "FB_Arithmetic add",
  "timeout": 5000,
  "continue_on_fail": false,
  "steps": [
    { "action": "write", "variable": "MAIN.fbArith.rA", "value": 10.0 },
    { "action": "write", "variable": "MAIN.fbArith.rB", "value": 3.0 },
    { "action": "wait", "ms": 200 },
    { "action": "read", "variable": "MAIN.fbArith.rResult", "expected": 13.0, "tolerance": 0.001 }
  ]
}

Steps¶

write¶

{ "action": "write", "variable": "MAIN.fbArith.rA", "value": 10.0 }

Writes one online variable. A JSON bool goes as TRUE/FALSE, a number as its text; a string is the IEC literal as written (WORD#5, E_Mode.Run).

wait¶

{ "action": "wait", "ms": 200 }

Waits for the requested number of milliseconds (default 100), cut short at the timeout.

read¶

{
  "action": "read",
  "variable": "MAIN.fbArith.rResult",
  "expected": 13.0,
  "tolerance": 0.001
}

Reads one online variable. If expectation fields are present, the step also asserts the value.

Supported expectation fields:

assert¶

{ "action": "assert", "variable": "MAIN.fbArith.xDone", "expected": true }

The same as read.

Values¶

A read gives the value as the IDE holds it: a BOOL as true/false, a number as a number, anything else as the IDE's text. expected compares a bool with a bool, a number within tolerance, and text ignoring case against the value or the IDE's spelling of it (E_Mode.Run).

FB Testability Rule¶

For online tests, MAIN must pass FB instance fields through to the FB call. Otherwise the PLC program overwrites values written by the CLI before the FB can use them.

Wrong:

fbArith(rA := 0.0, rB := 0.0, eOp := 0, xExecute := FALSE);

Correct:

fbArith(rA := fbArith.rA, rB := fbArith.rB,
        eOp := fbArith.eOp, xExecute := fbArith.xExecute);

Edge Triggers¶

Online writes are committed individually. The PLC may execute a cycle between two writes. If an FB uses a rising-edge trigger, a fast false -> true transition can be missed.

For FBs that are tested through the online API, prefer continuous execution:

IF xExecute THEN
    rResult := rA + rB;
    xDone := TRUE;
ELSE
    xDone := FALSE;
END_IF

If rising-edge logic is required, add an explicit wait between reset and trigger writes:

{ "action": "write", "variable": "MAIN.fbArith.xExecute", "value": false },
{ "action": "wait", "ms": 100 },
{ "action": "write", "variable": "MAIN.fbArith.xExecute", "value": true }

Result¶

The CLI returns structured JSON.

Typical result shape:

{
  "status": "SUCCESS",
  "ok": true,
  "summary": {
    "ok": 3,
    "not_ok": 0,
    "total": 3,
    "files": 2
  },
  "files": [
    {
      "file": "arithmetic.json",
      "plan": "Arithmetic smoke test",
      "status": "SUCCESS",
      "ok": true,
      "tests_ok": 2,
      "tests_failed": 0,
      "total_ms": 512
    }
  ],
  "results": []
}

results holds one report per plan: the tests, and per test its steps with status, the value read and the error of a failed step.

Common Failures¶