Automator not evaluating action / calculation

[Hormo]

I am trying to run verification cases with different sets of values (verification) over a part that performs a set of actions that themselves use calculations. Syside 0.8.4 does expose expression evaluation but not full execution of the verification action steps to concrete runtime values. Is this expected or did I do something wrong?
So the flows are modeled correctly and validated, but their runtime value propagation is not exposed as concrete execution outputs through this Syside Python API path.

PS: PassIf exists conceptually in VerificationCases, but is currently not evaluable via model-level expression evaluation in this execution path
PS: Looks like Syside does not execute flow or bind into concrete runtime values. So I have no way to runtime it other than rebuilding the calculations per attribute (that makes the whole calculation semantic useless)

I have been doing some testing and here are the results:

SysML v2 / Automator Runtime Materialization Gap

Summary

This post reports a repeatable gap between semantic validity and runtime value materialization when using SysML v2 verification cases through the Python Automator API.

In short:

• Models load and validate cleanly.
• Expression-level features evaluate correctly.
• Action/calc execution paths often remain symbolic instead of materializing primitive runtime values.

This makes it hard to claim full behavioral verification through verification cases when verdict logic depends on action/calc internals.

What was tested

A verification-case setup with:

• Scalar inputs
• Multiple actions
• calc usages
• assign, bind, flow, control flow (if, ordered actions)
• Returned verdict from verification case
• Evaluation via Compiler.evaluate_feature(...) and pipeline execution

Observed behavior

1. Model diagnostics

• Parse/semantic diagnostics can be zero.

2. Expression features

• Direct attribute expressions materialize to primitives (float, bool) reliably.

3. Action/calc paths

• Features from action/calc networks often evaluate to symbolic objects:
  • ReferenceUsage
  • AttributeUsage
  • ItemUsage
  • CalculationUsage
  • ActionUsage
• calc usage .result can remain symbolic in case scope.

4. Pattern variation

• Switching between action ... : Type and perform action ... defined by Type did not materially change this behavior in tested cases.

Why this matters

For executable verification governance:

• A semantically valid model does not necessarily provide materialized runtime values for behavioral paths.
• If verdict depends on action/calc internals, verification may not be executable in practice.
• Teams may fall back to expression-level “pseudo verification” as a workaround, which weakens confidence in full behavior execution claims.

Current workaround

Use expression-level verdict evidence (evaluable to primitive values) for execution/reporting pipelines, while keeping action/calc structure for model intent.

This is practical, but it does not fully substitute end-to-end behavioral materialization.

Improvement requests

1. Clarify execution contract

• Document exactly which modeling constructs are guaranteed to materialize runtime values under current Automator behavior.

2. Expose materialization status

• Provide explicit API signals for “materialized value” vs “symbolic model object”.

3. Strengthen action/calc execution

• Improve materialization for:
  • calc usage results
  • nested action outputs
  • assignment/binding-derived values
  • verification return paths derived from behavior networks

4. Publish recommended executable profile

• A concise “authoring profile” for verification cases that are expected to execute with concrete runtime outputs.

Suggested diagnostic artifact

A built-in materialization matrix per run:

• feature path
• scope
• returned value type
• materialized/symbolic classification
• diagnostics

This would make tool behavior transparent and easier to govern in CI pipelines.

[Daumantas]

@Hormo

Syside 0.8.4 does expose expression evaluation but not full execution of the verification action steps to concrete runtime values. Is this expected or did I do something wrong?

You are correct that Syside currently only supports evaluating some expressions. Full support is still a work-in-progress.

@Hormo

Document exactly which modeling constructs are guaranteed to materialize runtime values under current Automator behavior.

I believe this should be covered by Expression Evaluation. All Literal... elements are unwrapped because they are… literal, and their values are known without evaluation. Other than that, evaluation continues reducing expressions until only the result is left, or a fatal error was encountered.

@Hormo

Provide explicit API signals for “materialized value” vs “symbolic model object”.

We do not actually materialize anything as evidenced by returning literal values whenever possible. Any Elements returned are either because they were referenced through FeatureReferenceExpressions, or the evaluated Feature had no bound value.

@Hormo

Improve materialization for:

• calc usage results
• nested action outputs
• assignment/binding-derived values
• verification return paths derived from behavior networks

Next release has improves to expression evaluation, namely evaluating user-defined functions. Simulation (actions etc.) will have to wait still.

@Hormo

A built-in materialization matrix per run:

• feature path
• scope
• returned value type
• materialized/symbolic classification
• diagnostics

Do you have a concrete example of inputs/outputs?

[Hormo]

@Daumantas

Do you have a concrete example of inputs/outputs?

I can evaluate expression-level verification features, but action/calc-path internals remain symbolic objects instead of runtime primitive values.

This blocks full behavioral verification and forces a pseudo-verification workaround (verdict derived from expression attributes in the verification case definition).

Environment

• syside: 0.8.4
• OS: Windows
• Model load diagnostics: 0

Minimal Repro Shape

The verification case has:

• top-level expression attributes (e.g., reportDeltaOutcome)
• sub-actions inside processData (Exercise..., Compare...)
• calc usages inside sub-actions (adjNomCalc, deltaCalc, etc.)
• action outputs bound to evaluation

What I Evaluate

• Expression-level features in case scope
• Action/calc internals in case scope

Expected

• If action/calc execution is supported for this pattern, action/calc results should materialize to primitive values (or at least materialized item attribute values).

Actual

• Expression-level features materialize (float, bool)
• Action/calc internals stay symbolic (ActionUsage, CalculationUsage, ReferenceUsage, ItemUsage, AttributeUsage)
• No diagnostics are emitted for this condition

Concrete Output (from one case)

load_diagnostics 0
case.reportNetExerciseOutcome => float 600.0 fatal False diag 0
case.reportComparatorOutcome => float 549.9999999999999 fatal False diag 0
case.reportDeltaOutcome => float 50.000000000000114 fatal False diag 0
case.reportVerdictPass => bool True fatal False diag 0
case.verdict => bool True fatal False diag 0

case.processData.evaluation => ItemUsage <...> fatal False diag 0
case.processData.ex => ActionUsage <...> fatal False diag 0
case.processData.cmp => ActionUsage <...> fatal False diag 0

case.processData.evaluation.netExerciseOutcome => AttributeUsage <...> fatal False diag 0
case.processData.evaluation.creditEventOutcome => AttributeUsage <...> fatal False diag 0
case.processData.evaluation.deltaOutcome => AttributeUsage <...> fatal False diag 0
case.processData.evaluation.preferExercise => AttributeUsage <...> fatal False diag 0

case.processData.ex.adjNomCalc => CalculationUsage <...> fatal False diag 0
case.processData.ex.adjNomCalc.result => ReferenceUsage <...> fatal False diag 0
case.processData.ex.netExerciseCalc => CalculationUsage <...> fatal False diag 0
case.processData.ex.netExerciseCalc.result => ReferenceUsage <...> fatal False diag 0
case.processData.ex.result => ItemUsage <...> fatal False diag 0

case.processData.cmp.defNomCalc => CalculationUsage <...> fatal False diag 0
case.processData.cmp.defNomCalc.result => ReferenceUsage <...> fatal False diag 0
case.processData.cmp.deltaCalc => CalculationUsage <...> fatal False diag 0
case.processData.cmp.deltaCalc.result => ReferenceUsage <...> fatal False diag 0
case.processData.cmp.preferCalc => CalculationUsage <...> fatal False diag 0
case.processData.cmp.preferCalc.result => ReferenceUsage <...> fatal False diag 0
case.processData.cmp.evaluation => ItemUsage <...> fatal False diag 0
case.processData.cmp.creditResult => ItemUsage <...> fatal False diag 0

Additional Check: PassIf(...)

In this environment:

• PassIf(...) parses
• but evaluation fails with:
  • error (model-error): Function 'PassIf' is not a model-level evaluable function

[Daumantas]

@Hormo

What I Evaluate

• Expression-level features in case scope
• Action/calc internals in case scope

Expected

• If action/calc execution is supported for this pattern, action/calc results should materialize to primitive values (or at least materialized item attribute values).

Calculation invocations will work in the next release. Actions will not, at least for a while.

Note that evaluating references is not an error in itself:

attribute a { attribute x = 4; }
attribute b = a;
attribute c = b.x;

Evaluation of b returns a as it is needed to implement member access when evaluating c.

@Hormo

In this environment:

• PassIf(...) parses
• but evaluation fails with:
  • error (model-error): Function 'PassIf' is not a model-level evaluable function

This will work in the next release.