Minimal JSON round-trip with external/library references produces unresolved-reference warnings and non-identical derived relationships after reconstruction

[Robert_Karban]

Summary:

When a model is exported to minimal JSON with include_cross_ref_uris=False, then re-imported with syside.json.loads(…), models that reference stdlib/library
elements emit unresolved-reference warnings during deserialization.

A stronger recovery path:

1. syside.json.loads(…)
2. DeserializedModel.link(IdMap of default environment docs)
3. syside.Sema().resolve(…)
4. syside.serialize(…, full options)

recovers the model much more accurately, including implied relationships and total element counts. However, the reconstructed full JSON is still not always
identical to the original full export from the text-loaded model. The remaining differences are localized in derived relationship fields such as feature,
featureMembership, inheritedFeature, usage, and some Subsetting targets.

Minimal reproducer for unresolved-reference warnings:

import warnings
import syside

src = “package P { private import ScalarValues::*; }”

model, diagnostics = syside.load_model(sysml_source=src)
assert not diagnostics.contains_errors(warnings_as_errors=True)

json_options = syside.JsonStringOptions()
json_options.include_cross_ref_uris = False
json_options.indent = False
writer = syside.JsonStringWriter(json_options)

options = syside.SerializationOptions.minimal()

with model.user_docs[0].lock() as locked:
syside.serialize(locked.root_node, writer, options)
raw_min = writer.result

with warnings.catch_warnings(record=True) as wlist:
warnings.simplefilter(“always”)
deserialized_model, _ = syside.json.loads(raw_min, “memory:///import.sysml”)

print([str(w.message) for w in wlist])

Minimal reproducer for external typed feature warning:

import warnings
import syside

src = “package P { part x { attribute mass :> ISQBase::mass; } }”

model, diagnostics = syside.load_model(sysml_source=src)
assert not diagnostics.contains_errors(warnings_as_errors=True)

json_options = syside.JsonStringOptions()
json_options.include_cross_ref_uris = False
json_options.indent = False
writer = syside.JsonStringWriter(json_options)

options = syside.SerializationOptions.minimal()

with model.user_docs[0].lock() as locked:
syside.serialize(locked.root_node, writer, options)
raw_min = writer.result

with warnings.catch_warnings(record=True) as wlist:
warnings.simplefilter(“always”)
deserialized_model, _ = syside.json.loads(raw_min, “memory:///import.sysml”)

print([str(w.message) for w in wlist])

Observed facts:

• The unresolved IDs are external/library elements, not missing local model elements.
• These IDs do exist in Environment.get_default().
• Examples observed:
  • 40bb440c-… → ScalarValues
  • 2a413ac9-… → SI
  • 95bd17de-… → USCustomaryUnits
  • 9af08108-… → ISQBase::mass

Recovery path that improves fidelity:

import syside

env = syside.Environment.get_default()

id_map = syside.IdMap()
for mutex in env.documents:
with mutex.lock() as dep:
id_map.insert_or_assign(dep)

deserialized_model.link(id_map)
syside.Sema().resolve([deserialized_model.document], env.index(), env.lib)

Actual behavior:

• syside.json.loads(…) emits unresolved-reference warnings when minimal JSON contains external/library references serialized without cross-ref URIs.
• The link(IdMap) + Sema.resolve(…) workaround reconstructs implied relationships and usually restores overall structure much better.
• But reconstructed full JSON can still differ from the original full export from the text-loaded model.

Real-model evidence:

Using Flashlight.sysml:

• Original full export and reconstructed export had:

  • same total element count: 2366
  • same implied element count: 671
  • same @id set and order

• Remaining differences were limited to 13 type/field combinations

• The differences were localized to derived relationship fields, including:

  • one PartUsage differing in:

    • feature
    • featureMembership
    • inheritedFeature
    • usage

  • one Subsetting differing in:

    • general
    • subsettedFeature
    • relatedElement
    • target

Expected behavior:

One of these should be true:

1. JSON exported with include_cross_ref_uris=False should be documented as not cleanly round-trippable for models with external/library references.
2. syside.json.loads(…) should avoid unresolved-reference warnings for references that are expected to be resolved later from the default environment.
3. The combination of JSON deserialization, environment linking, and semantic resolution should reconstruct the same derived relationship bindings as the
   original text-loaded model, if minimal JSON is intended to be a lossless model persistence format.

Likely root cause:

• With include_cross_ref_uris=False, external references are serialized as bare @id references.
• During json.loads(…), these become pending references with empty URI.
• This is not enough information for deserialization-time resolution.
• Later environment-based linking and semantic resolution recover most of the structure, but some derived relationships appear to be rebound differently from
  the original text-loaded model.

Why this matters:

Minimal JSON appears to work as a compact persistence format, but for models with external/library references it is currently:

• noisy at deserialization time due to warnings
• not fully stable under round-trip reconstruction, even with environment linking and semantic resolution

Flashlight.sysml (19.6 KB)

[Daumantas]

Hi,

This is the intended behaviour for single JSON deserialization as we cannot know whether the user intended to resolve references from multiple, potentially cyclically dependent, sources or not. Documentation of return value of json.loads clearly states that. That is why unresolved references are only warnings and not errors.

Note that there is also a project deserialization overload json.loads that accepts an iterable of tuple[str | Url | Document, str] which will resolve pending references after deserialization.

[Robert_Karban]

Hi @Daumantas,

I applied the recommended recovery path in our wrapper:

1. project-style syside.json.loads([...])
2. DeserializedModel.link(...) with default-environment docs in the IdMap
3. syside.Sema().resolve(...)
4. then either pretty-print to text or serialize to full JSON

Results:

• This fixes the text reconstruction issue from topic 512 in our round-trip path.
• It also removes the unresolved external/library reference warnings we were seeing when reconstructing text from minimal JSON.
• For many models, full JSON → text → full JSON → text is now stable after one printer pass.

However, we still see remaining gaps for model-based minimal → full JSON reconstruction:

• Flashlight.sysml
• multi-file Flashlight
• Test5.sysml
• Test7.sysml
  like:
• feature
• featureMembership
• inheritedFeature
• usage
• some Subsetting targets / related fields

For Test5 / Test7, reconstructed text can also fail reparsing with metadata-related errors such as:

• metadata-feature-annotated-element: Metadata feature cannot annotate Usage
• incomplete semantic metadata / unresolved baseType diagnostics

So at this point the sema-recovery path clearly improves fidelity a lot, but it does not yet make minimal JSON reconstruction fully equivalent to direct full export for all models.

import json
import syside

def expand_minimal_json_to_full_json(document_sources):
"""
document_sources:
iterable of (url, json_string), one per root document
example: [("memory:///Flashlight.sysml", raw_min_json)]
"""
_project_model, deserialized_results = syside.json.loads(document_sources)

  env = syside.Environment.get_default()
  id_map = syside.IdMap()

  for mutex in env.documents:
      with mutex.lock() as dep:
          id_map.insert_or_assign(dep)

  docs_to_resolve = []
  for deserialized_model, _report in deserialized_results:
      docs_to_resolve.append(deserialized_model.document)
      with deserialized_model.document.mutex.lock() as locked_document:
          id_map.insert_or_assign(locked_document)

  for deserialized_model, _report in deserialized_results:
      deserialized_model.link(id_map)

  syside.Sema().resolve(
      docs_to_resolve,
      env.index(),
      env.lib,
  )

  json_options = syside.JsonStringOptions()
  json_options.include_cross_ref_uris = False
  json_options.indent = False

  options = syside.SerializationOptions().minimal().with_options(
      use_standard_names=True,
      include_derived=True,
      include_redefined=True,
      include_default=False,
      include_optional=False,
      include_implied=True,
  )
  options.fail_action = syside.FailAction.Ignore

  full_documents = []
  for deserialized_model, _report in deserialized_results:
      writer = syside.JsonStringWriter(json_options)
      with deserialized_model.document.mutex.lock() as locked_document:
          syside.serialize(locked_document.root_node, writer, options)
      full_documents.append(json.loads(writer.result))

  return full_documents

For JSON -> text reconstruction:

import syside

def json_to_sysml_text(json_string):
_project_model, deserialized_results = syside.json.loads(
[("memory:///import.sysml", json_string)]
)
deserialized_model, _report = deserialized_results[0]

  env = syside.Environment.get_default()
  id_map = syside.IdMap()

  for mutex in env.documents:
      with mutex.lock() as dep:
          id_map.insert_or_assign(dep)

  with deserialized_model.document.mutex.lock() as locked_document:
      id_map.insert_or_assign(locked_document)

  deserialized_model.link(id_map)

  syside.Sema().resolve(
      [deserialized_model.document],
      env.index(),
      env.lib,
  )

  printer_cfg = syside.PrinterConfig(line_width=80, tab_width=2)
  printer = syside.ModelPrinter.sysml()
  return syside.pprint(deserialized_model.document.root_node, printer, printer_cfg)

Is there a newer recommended reconstruction path for achieving byte-equivalent full JSON, or should these remaining cases still be considered known limitations of minimal JSON round-
trip?

Thanks,

Robert

2_FlashlightStarterModel.sysml (7.9 KB)

1_FlashlightContextClassExercise.sysml (10.6 KB)

3_GeneralConcepts.sysml (1.1 KB)

Flashlight.sysml (19.6 KB)

Test7.sysml (72.7 KB)

Test5.sysml (72.7 KB)

[Daumantas]

@Robert_Karban

• metadata-feature-annotated-element: Metadata feature cannot annotate Usage

Printer does not insert connection keyword so the element becomes a Usage instead.

@Robert_Karban

• incomplete semantic metadata / unresolved baseType diagnostics

Sema on synthetic elements (no source) does not eagerly resolve metadata elements.

I will have a look into both of those issues.

[Daumantas]

Fixed for the next release.