Automator not evaluating action / calculation

Syside
1

I am trying to run verification cases with different sets of values (verification) over a part that performs a set of actions that themselves use calculations. Syside 0.8.4 does expose expression evaluation but not full execution of the verification action steps to concrete runtime values. Is this expected or did I do something wrong?
So the flows are modeled correctly and validated, but their runtime value propagation is not exposed as concrete execution outputs through this Syside Python API path.

PS: PassIf exists conceptually in VerificationCases, but is currently not evaluable via model-level expression evaluation in this execution path
PS: Looks like Syside does not execute flow or bind into concrete runtime values. So I have no way to runtime it other than rebuilding the calculations per attribute (that makes the whole calculation semantic useless)

I have been doing some testing and here are the results:

SysML v2 / Automator Runtime Materialization Gap

Summary

This post reports a repeatable gap between semantic validity and runtime value materialization when using SysML v2 verification cases through the Python Automator API.

In short:

  • Models load and validate cleanly.
  • Expression-level features evaluate correctly.
  • Action/calc execution paths often remain symbolic instead of materializing primitive runtime values.

This makes it hard to claim full behavioral verification through verification cases when verdict logic depends on action/calc internals.

What was tested

A verification-case setup with:

  • Scalar inputs
  • Multiple actions
  • calc usages
  • assign, bind, flow, control flow (if, ordered actions)
  • Returned verdict from verification case
  • Evaluation via Compiler.evaluate_feature(...) and pipeline execution

Observed behavior

  1. Model diagnostics
  • Parse/semantic diagnostics can be zero.
  1. Expression features
  • Direct attribute expressions materialize to primitives (float, bool) reliably.
  1. Action/calc paths
  • Features from action/calc networks often evaluate to symbolic objects:
    • ReferenceUsage
    • AttributeUsage
    • ItemUsage
    • CalculationUsage
    • ActionUsage
  • calc usage .result can remain symbolic in case scope.
  1. Pattern variation
  • Switching between action ... : Type and perform action ... defined by Type did not materially change this behavior in tested cases.

Why this matters

For executable verification governance:

  • A semantically valid model does not necessarily provide materialized runtime values for behavioral paths.
  • If verdict depends on action/calc internals, verification may not be executable in practice.
  • Teams may fall back to expression-level “pseudo verification” as a workaround, which weakens confidence in full behavior execution claims.

Current workaround

Use expression-level verdict evidence (evaluable to primitive values) for execution/reporting pipelines, while keeping action/calc structure for model intent.

This is practical, but it does not fully substitute end-to-end behavioral materialization.

Improvement requests

  1. Clarify execution contract
  • Document exactly which modeling constructs are guaranteed to materialize runtime values under current Automator behavior.
  1. Expose materialization status
  • Provide explicit API signals for “materialized value” vs “symbolic model object”.
  1. Strengthen action/calc execution
  • Improve materialization for:
    • calc usage results
    • nested action outputs
    • assignment/binding-derived values
    • verification return paths derived from behavior networks
  1. Publish recommended executable profile
  • A concise “authoring profile” for verification cases that are expected to execute with concrete runtime outputs.

Suggested diagnostic artifact

A built-in materialization matrix per run:

  • feature path
  • scope
  • returned value type
  • materialized/symbolic classification
  • diagnostics

This would make tool behavior transparent and easier to govern in CI pipelines.

2

You are correct that Syside currently only supports evaluating some expressions. Full support is still a work-in-progress.

I believe this should be covered by Expression Evaluation. All Literal... elements are unwrapped because they are… literal, and their values are known without evaluation. Other than that, evaluation continues reducing expressions until only the result is left, or a fatal error was encountered.

We do not actually materialize anything as evidenced by returning literal values whenever possible. Any Elements returned are either because they were referenced through FeatureReferenceExpressions, or the evaluated Feature had no bound value.

Next release has improves to expression evaluation, namely evaluating user-defined functions. Simulation (actions etc.) will have to wait still.

Do you have a concrete example of inputs/outputs?

3

I can evaluate expression-level verification features, but action/calc-path internals remain symbolic objects instead of runtime primitive values.

This blocks full behavioral verification and forces a pseudo-verification workaround (verdict derived from expression attributes in the verification case definition).

Environment

  • syside: 0.8.4
  • OS: Windows
  • Model load diagnostics: 0

Minimal Repro Shape

The verification case has:

  • top-level expression attributes (e.g., reportDeltaOutcome)
  • sub-actions inside processData (Exercise..., Compare...)
  • calc usages inside sub-actions (adjNomCalc, deltaCalc, etc.)
  • action outputs bound to evaluation

What I Evaluate

  • Expression-level features in case scope
  • Action/calc internals in case scope

Expected

  • If action/calc execution is supported for this pattern, action/calc results should materialize to primitive values (or at least materialized item attribute values).

Actual

  • Expression-level features materialize (float, bool)
  • Action/calc internals stay symbolic (ActionUsage, CalculationUsage, ReferenceUsage, ItemUsage, AttributeUsage)
  • No diagnostics are emitted for this condition

Concrete Output (from one case)

load_diagnostics 0
case.reportNetExerciseOutcome => float 600.0 fatal False diag 0
case.reportComparatorOutcome => float 549.9999999999999 fatal False diag 0
case.reportDeltaOutcome => float 50.000000000000114 fatal False diag 0
case.reportVerdictPass => bool True fatal False diag 0
case.verdict => bool True fatal False diag 0

case.processData.evaluation => ItemUsage <...> fatal False diag 0
case.processData.ex => ActionUsage <...> fatal False diag 0
case.processData.cmp => ActionUsage <...> fatal False diag 0

case.processData.evaluation.netExerciseOutcome => AttributeUsage <...> fatal False diag 0
case.processData.evaluation.creditEventOutcome => AttributeUsage <...> fatal False diag 0
case.processData.evaluation.deltaOutcome => AttributeUsage <...> fatal False diag 0
case.processData.evaluation.preferExercise => AttributeUsage <...> fatal False diag 0

case.processData.ex.adjNomCalc => CalculationUsage <...> fatal False diag 0
case.processData.ex.adjNomCalc.result => ReferenceUsage <...> fatal False diag 0
case.processData.ex.netExerciseCalc => CalculationUsage <...> fatal False diag 0
case.processData.ex.netExerciseCalc.result => ReferenceUsage <...> fatal False diag 0
case.processData.ex.result => ItemUsage <...> fatal False diag 0

case.processData.cmp.defNomCalc => CalculationUsage <...> fatal False diag 0
case.processData.cmp.defNomCalc.result => ReferenceUsage <...> fatal False diag 0
case.processData.cmp.deltaCalc => CalculationUsage <...> fatal False diag 0
case.processData.cmp.deltaCalc.result => ReferenceUsage <...> fatal False diag 0
case.processData.cmp.preferCalc => CalculationUsage <...> fatal False diag 0
case.processData.cmp.preferCalc.result => ReferenceUsage <...> fatal False diag 0
case.processData.cmp.evaluation => ItemUsage <...> fatal False diag 0
case.processData.cmp.creditResult => ItemUsage <...> fatal False diag 0

Additional Check: PassIf(...)

In this environment:

  • PassIf(...) parses
  • but evaluation fails with:
    • error (model-error): Function 'PassIf' is not a model-level evaluable function
4

Calculation invocations will work in the next release. Actions will not, at least for a while.

Note that evaluating references is not an error in itself:

attribute a { attribute x = 4; }
attribute b = a;
attribute c = b.x;

Evaluation of b returns a as it is needed to implement member access when evaluating c.

This will work in the next release.

1 Like
5

Automator Repro: Action/Calc Paths Stay Symbolic While Expression Features Materialize

New attempt: Summary

I can evaluate expression-level verification features, but action/calc-path internals remain symbolic objects instead of runtime primitive values.

This blocks full behavioral verification and forces a pseudo-verification workaround (verdict derived from expression attributes in the verification case definition).

Environment

  • Retested on 2026-04-17

  • syside: 0.8.7

  • OS: Windows

  • Model load diagnostics: 436 for the full project model load used by the script

Current Status in 0.8.7

  • The core repro still stands: expression-level verification features materialize, but action/calc-path internals remain symbolic.

Change Since Earlier Check

  • In the earlier 0.8.4 check, PassIf(...) parsed but failed evaluation with Function 'PassIf' is not a model-level evaluable function.

  • In the current 0.8.7 retest, that specific PassIf(...) failure is no longer the active issue for this repro file.

  • The remaining issue is the lack of runtime materialization for nested action/calc internals.

Additional Check: PassIf(...)

Historical note:

  • In the earlier 0.8.4 environment, PassIf(...) parsed

  • but evaluation failed with:

  • error (model-error): Function 'PassIf' is not a model-level evaluable function

Current retest in 0.8.7:

  • This specific PassIf(...) evaluation failure is not what reproduces for StandaloneExerciseDecisionTestCases.sysml

  • The persistent issue is symbolic action/calc internals during evaluation

Questions

  1. Is this symbolic behavior for action/calc internals expected in current releases?

  2. Is there a roadmap ETA for action/calc runtime materialization in Automator evaluation?

  3. Could the API expose an explicit materialized-vs-symbolic status per evaluated feature?

6

This happens if features in expressions do not have primitive values assigned to them.

Do you have a concrete example? It is hard to tell just from a few statements what is going wrong.

Note that PassIf returns a verdict enum - we cannot convert it to a primitive value because there is no primitive value! You can do postprocessing based on the returned element .name.

Yes, not all expressions can evaluate to primitive values. One critical example is semantic metadata which must evaluate to a bound metadata feature so that we can insert an implicit specialization during sema.

Everything that can be evaluated as a primitive value bar missing types (e.g. complex) is already evaluated as one.

For single values, isinstance(result, syside.Element) is enough. For lists, apply isinstance to every value instead.