Json serialization/deserialization is not symmetric

Robert_Karban · September 16, 2025, 10:25pm

Hi,

When using

syside.SerializationOptions.minimal()

I can parse text to json, and json to text without problems.

When I use:
options = syside.SerializationOptions()

options = options.with_options(

    use_standard_names=True,

    include_derived=True,

    include_redefined=True,

    include_default=False,

    include_optional=False,

    include_implied=True,

)

I can serialize but the deserialization fails.

Thanks,

Robert

Daumantas · September 18, 2025, 9:45am

Hi Robert,

can you provide a SysML snippet that triggers deserialization failure?

Robert_Karban · September 18, 2025, 10:47am

Hi @Daumantas ,

Thank you for responding.

Certainly! I discussed it already in depth with Simonas.

This is the official request now

It’s literally any sysml textual model.

Below are the current code and examples. I’m only allowed to post one link…

Currently a lot of post processing is necessary to make the json parsable and there are still gaps.

Cleanup code and explanation you can find here:

github.com/Open-MBEE/flexo_syside

src/flexo_syside_lib/utils.py

main

import json
from typing import Any, Dict, List, Set, Tuple
import re
import difflib

def clean_sysml_json_for_syside(
    json_text: str,
    *,
    allow_python_literals: bool = False,
    indent: int = 2,
    preserve_refs_with_uri: bool = True,
) -> str:
    """
    Clean a SysMLv2 JSON document for syside deserialization:
      1) Remove dangling thin references (objects that only have @-prefixed keys with @id not defined).
      2) Drop relationship elements whose required related elements are missing after step 1.
      3) Iterate to a fixpoint.

    Relationship integrity rules (best-effort; extends as needed):
      - Subsetting: requires both ends, via either {"specific","general"} or {"subsettingFeature","subsettedFeature"}.

This file has been truncated. show original

any help will be greatly appreciated and move sysmlv2 and syside forward a lot.

The example notebook shows how to use the library

Daumantas · September 18, 2025, 12:20pm

package P1 {
    part p1 {
        attribute a1 = 9999;
        attribute a2 = a1 + 1;
    }
}

roundtrips just fine with

syside.SerializationOptions.minimal().with_options(
    use_standard_names=True,
    include_derived=True,
    include_redefined=True,
    include_default=False,
    include_optional=False,
    include_implied=True,
)

Note that we run roundtrip tests for both minimal and full serialization options. Have you compared JSONs used in deserialization?

Robert_Karban · September 18, 2025, 2:16pm

Hm. Interesting.

Maybe the deserialization code is wrong?

See function convert_json_to_sysml_textual()

If i don’t run clean_sysml_json_for_ syside (), models won’t deserialize. I had a long discussion with simonas about that.

It’s because of dangling references. I compared thr json yes.

I’ll post an example that doesn’t work in a bit.

Robert_Karban · September 18, 2025, 4:03pm

Hi @Daumantas ,

here’s a snippet that doesn’t deserialize, even when you take out all the standard library references:

library package A {

private import ScalarValues::\*;

private import ISQ::\*;

private import SI::\*;



attribute def Orbit {

    attribute semiMajorAxis:> ISQ::length = 7000 \[SI::km\];

    attribute eccentricity: ScalarValues::Real = 0.01;

    attribute inclination:ScalarValues::Real = 28.5 \[SI::degree\] {

        assert constraint { that >= 0.0 and that <= 90.0 }

    }



    attribute RAAN:ScalarValues::Real = 0.0 \[SI::degree\];  // Right Ascension of Ascending Node

    attribute argumentOfPeriapsis:ScalarValues::Real = 0.0 \[SI::degree\];

    attribute trueAnomaly:ScalarValues::Real = 0.0 \[SI::degree\];

}

}

library package SDK {

private import ScalarValues::\*;

private import ISQ::\*;

private import SI::\*;

private import A::\*;

// structure

port def P;



part def S {

    port scp : P;

    attribute uid;

    attribute typeID;

}



part def N {

    port scp_outside2 : P;

    attribute uid;

}



part def M {

    attribute uid;

    part n: N\[0..\*\];

    part s: S\[0..\*\];

}



part def T :> S {

    attribute :>> typeID = 2;

}

}

package M1 {

private import SDK::\*;



part m0001_2N : M {

    attribute :>> uid = "M0001_2N";

    part nx0001 : N subsets n {

        attribute :>> uid = "NX0001";

    }

    part tcs0001 : T subsets s {

        attribute :>> uid = "TCS0001";

    }



    interface tcs0001.scp to nx0001.scp_outside2;

}

}

Errors:

[Warning] Could not find reference to “40bb440c-5036-58e1-8675-5afccb8b8f1d”
02. [Error] No element 40bb440c-5036-58e1-8675-5afccb8b8f1d found
03. [Warning] Could not find reference to “3fcc2ef4-a31c-522b-b69d-717d74bccfa6”
04. [Error] No element 3fcc2ef4-a31c-522b-b69d-717d74bccfa6 found
05. [Warning] Could not find reference to “2a413ac9-972d-5182-afd3-46cdde4e0eba”
06. [Error] No element 2a413ac9-972d-5182-afd3-46cdde4e0eba found
07. [Warning] Could not find reference to “04e5e2ff-9907-5f48-a055-47bba44803af”
08. [Error] No element 04e5e2ff-9907-5f48-a055-47bba44803af found
09. [Warning] Could not find reference to “04e5e2ff-9907-5f48-a055-47bba44803af”
10. [Error] No element 04e5e2ff-9907-5f48-a055-47bba44803af found
11. [Warning] Could not find reference to “95649c90-b777-51a9-bb8d-2116326159ed”
12. [Error] No element 95649c90-b777-51a9-bb8d-2116326159ed found
13. [Error] Invalid namespace body relationship - missing related element
14. [Error] Invalid heritage relationship - missing related element
15. [Warning] Could not find reference to “14c0aa22-5489-59b5-b438-ded26e83ba31”
16. [Error] No element 14c0aa22-5489-59b5-b438-ded26e83ba31 found
17. [Warning] Could not find reference to “14c0aa22-5489-59b5-b438-ded26e83ba31”
18. [Error] No element 14c0aa22-5489-59b5-b438-ded26e83ba31 found
19. [Error] Invalid heritage relationship - missing related element
20. [Warning] Could not find reference to “14c0aa22-5489-59b5-b438-ded26e83ba31”
21. [Error] No element 14c0aa22-5489-59b5-b438-ded26e83ba31 found
22. [Warning] Could not find reference to “14c0aa22-5489-59b5-b438-ded26e83ba31”
23. [Error] No element 14c0aa22-5489-59b5-b438-ded26e83ba31 found
24. [Warning] Could not find reference to “54bd1450-5a8b-550f-b4c2-3bc918783043”
25. [Error] No element 54bd1450-5a8b-550f-b4c2-3bc918783043 found
26. [Error] Invalid namespace body relationship - missing related element
27. [Warning] Could not find reference to “dbd75b49-0e57-5cf9-ba35-9f8843b1e657”
28. [Error] No element dbd75b49-0e57-5cf9-ba35-9f8843b1e657 found
29. [Error] Invalid namespace body relationship - missing related element
30. [Warning] Could not find reference to “dbd75b49-0e57-5cf9-ba35-9f8843b1e657”
31. [Error] No element dbd75b49-0e57-5cf9-ba35-9f8843b1e657 found
32. [Error] Invalid namespace body relationship - missing related element
33. [Error] Invalid heritage relationship - missing related element
34. [Warning] Could not find reference to “14c0aa22-5489-59b5-b438-ded26e83ba31”
35. [Error] No element 14c0aa22-5489-59b5-b438-ded26e83ba31 found
36. [Warning] Could not find reference to “14c0aa22-5489-59b5-b438-ded26e83ba31”
37. [Error] No element 14c0aa22-5489-59b5-b438-ded26e83ba31 found
38. [Warning] Could not find reference to “54bd1450-5a8b-550f-b4c2-3bc918783043”
39. [Error] No element 54bd1450-5a8b-550f-b4c2-3bc918783043 found
40. [Error] Invalid namespace body relationship - missing related element
41. [Error] Invalid heritage relationship - missing related element
42. [Warning] Could not find reference to “14c0aa22-5489-59b5-b438-ded26e83ba31”
43. [Error] No element 14c0aa22-5489-59b5-b438-ded26e83ba31 found
44. [Warning] Could not find reference to “14c0aa22-5489-59b5-b438-ded26e83ba31”
45. [Error] No element 14c0aa22-5489-59b5-b438-ded26e83ba31 found
46. [Warning] Could not find reference to “54bd1450-5a8b-550f-b4c2-3bc918783043”
47. [Error] No element 54bd1450-5a8b-550f-b4c2-3bc918783043 found
48. [Error] Invalid namespace body relationship - missing related element
49. [Error] Invalid heritage relationship - missing related element
50. [Warning] Could not find reference to “14c0aa22-5489-59b5-b438-ded26e83ba31”
51. [Error] No element 14c0aa22-5489-59b5-b438-ded26e83ba31 found
52. [Warning] Could not find reference to “14c0aa22-5489-59b5-b438-ded26e83ba31”
53. [Error] No element 14c0aa22-5489-59b5-b438-ded26e83ba31 found
54. [Warning] Could not find reference to “54bd1450-5a8b-550f-b4c2-3bc918783043”
55. [Error] No element 54bd1450-5a8b-550f-b4c2-3bc918783043 found
56. [Error] Invalid namespace body relationship - missing related element
57. [Error] Invalid heritage relationship - missing related element
58. [Error] Invalid namespace body relationship - missing related element
59. [Error] Invalid namespace body relationship - missing related element
60. [Error] Invalid namespace body relationship - missing related element
61. [Warning] Could not find reference to “40bb440c-5036-58e1-8675-5afccb8b8f1d”
62. [Error] No element 40bb440c-5036-58e1-8675-5afccb8b8f1d found
63. [Warning] Could not find reference to “3fcc2ef4-a31c-522b-b69d-717d74bccfa6”
64. [Error] No element 3fcc2ef4-a31c-522b-b69d-717d74bccfa6 found
65. [Warning] Could not find reference to “2a413ac9-972d-5182-afd3-46cdde4e0eba”
66. [Error] No element 2a413ac9-972d-5182-afd3-46cdde4e0eba found
67. [Warning] Could not find reference to “3cfd22ed-a3cb-567d-b426-2d83d58a3fc5”
68. [Error] No element 3cfd22ed-a3cb-567d-b426-2d83d58a3fc5 found
69. [Error] Invalid namespace body relationship - missing related element
70. [Error] Invalid namespace body relationship - missing related element
71. [Error] Invalid namespace body relationship - missing related element
72. [Error] Invalid namespace body relationship - missing related element

Robert_Karban · September 18, 2025, 6:09pm

Same model without standard libriaries - fails also.

library package A {

attribute def Orbit {

    attribute semiMajorAxis = 7000;

    attribute eccentricity = 0.01;

    attribute inclination = 28.5 {

        assert constraint { that >= 0.0 and that <= 90.0 }

    }



    attribute RAAN = 0.0;  // Right Ascension of Ascending Node

    attribute argumentOfPeriapsis = 0.0;

    attribute trueAnomaly = 0.0;

}

}

library package SDK {

private import A::\*;

// structure

port def P;



part def S {

    port scp : P;

    attribute uid;

    attribute typeID;

}



part def N {

    port scp_outside2 : P;

    attribute uid;

}



part def M {

    attribute uid;

    part n: N\[0..\*\];

    part s: S\[0..\*\];

}



part def T :> S {

    attribute :>> typeID = 2;

}

}

package M1 {

private import SDK::\*;



part m0001_2N : M {

    attribute :>> uid = "M0001_2N";

    part nx0001 : N subsets n {

        attribute :>> uid = "NX0001";

    }

    part tcs0001 : T subsets s {

        attribute :>> uid = "TCS0001";

    }



    interface tcs0001.scp to nx0001.scp_outside2;

}

}

Deserialization failed. Diagnostic report: 01. [Warning] Could not find reference to “dbd75b49-0e57-5cf9-ba35-9f8843b1e657” 02. [Error] No element dbd75b49-0e57-5cf9-ba35-9f8843b1e657 found 03. [Error] Invalid namespace body relationship - missing related element 04. [Warning] Could not find reference to “dbd75b49-0e57-5cf9-ba35-9f8843b1e657” 05. [Error] No element dbd75b49-0e57-5cf9-ba35-9f8843b1e657 found 06. [Error] Invalid namespace body relationship - missing related element

cc: @rubayet

Daumantas · September 19, 2025, 6:40am

Still cannot reproduce with

import difflib
import syside

SRC0 = """\
library package A {
    attribute def Orbit {
        attribute semiMajorAxis = 7000;

        attribute eccentricity = 0.01;

        attribute inclination = 28.5 {
            assert constraint { that >= 0.0 and that <= 90.0 }
        }

        attribute RAAN = 0.0; // Right Ascension of Ascending Node
        attribute argumentOfPeriapsis = 0.0;
        attribute trueAnomaly = 0.0;
    }
}

library package SDK {
    private import A::*;

    // structure

    port def P;

    part def S {
        port scp : P;
        attribute uid;
        attribute typeID;
    }

    part def N {
        port scp_outside2 : P;
        attribute uid;
    }

    part def M {
        attribute uid;
        part n : N [0..*];
        part s : S [0..*];
    }

    part def T :> S {
        attribute :>> typeID = 2;
    }
}

package M1 {
    private import SDK::*;

    part m0001_2N : M {
        attribute :>> uid = "M0001_2N";

        part nx0001 : N subsets n {
            attribute :>> uid = "NX0001";
        }

        part tcs0001 : T subsets s {
            attribute :>> uid = "TCS0001";
        }

        interface tcs0001.scp to nx0001.scp_outside2;
    }
}
"""

SRC1 = """\
library package A {
    private import ScalarValues::*;
    private import ISQ::*;
    private import SI::*;

    attribute def Orbit {
        attribute semiMajorAxis :> ISQ::length = 7000 [SI::km];

        attribute eccentricity : ScalarValues::Real = 0.01;

        attribute inclination : ScalarValues::Real = 28.5 [SI::degree] {
            assert constraint { that >= 0.0 and that <= 90.0 }
        }

        attribute RAAN : ScalarValues::Real = 0.0 [SI::degree]; // Right Ascension of Ascending Node
        attribute argumentOfPeriapsis : ScalarValues::Real = 0.0 [SI::degree];
        attribute trueAnomaly : ScalarValues::Real = 0.0 [SI::degree];
    }
}

library package SDK {
    private import ScalarValues::*;
    private import ISQ::*;
    private import SI::*;
    private import A::*;

    // structure

    port def P;

    part def S {
        port scp : P;
        attribute uid;
        attribute typeID;
    }

    part def N {
        port scp_outside2 : P;
        attribute uid;
    }

    part def M {
        attribute uid;
        part n : N [0..*];
        part s : S [0..*];
    }

    part def T :> S {
        attribute :>> typeID = 2;
    }
}

package M1 {
    private import SDK::*;

    part m0001_2N : M {
        attribute :>> uid = "M0001_2N";

        part nx0001 : N subsets n {
            attribute :>> uid = "NX0001";
        }

        part tcs0001 : T subsets s {
            attribute :>> uid = "TCS0001";
        }

        interface tcs0001.scp to nx0001.scp_outside2;
    }
}
"""

SRC = SRC1

model, _ = syside.load_model(sysml_source=SRC)

with model.user_docs[0].lock() as doc:
    root_node = doc.root_node

string = syside.json.dumps(
    root_node,
    syside.SerializationOptions.minimal().with_options(
        use_standard_names=True,
        include_derived=True,
        include_redefined=True,
        include_default=False,
        include_optional=False,
        include_implied=True,
    ),
)

des, _ = syside.json.loads(string, "memory:///test.sysml")

map = syside.IdMap()
for mutex in model.environment.documents:
    with mutex.lock() as dep:
        map.insert_or_assign(dep)

report, success = des.link(map)
assert success, str(report.messages)

input = syside.sexp(root_node, print_references=True)
output = syside.sexp(des.root, print_references=True)

print(
    "Diff:\n",
    "\n".join(
        difflib.unified_diff(
            input.splitlines(), output.splitlines(), "input.txt", "output.txt"
        )
    ),
)

(at least include_implied=True needed to ensure that models are identical). In all cases, diff is empty.

Robert_Karban · September 19, 2025, 12:54pm

Hi @Daumantas

Thank you.

You use the the data structure ‘model’, that you get from the serialization when loading the textual model,
in the deserialization. It makes the deserialization dependent on the serialization.
That’s the difference I found wrt my code.

So, if I don’t have access to ‘model’ that you create here:
model, _ = syside.load_model(sysml_source=SRC)
but just have the code below, how do I get it in order to build the map?

des, _ = syside.json.loads(string, “memory:///test.sysml”)

map = syside.IdMap()

for mutex in model.environment.documents:

with mutex.lock() as dep:

map.insert_or_assign(dep)

report, success = des.link(map)

Daumantas · September 19, 2025, 2:00pm

The only dependency in the deserialization is from model.environment which in this case is the bundled standard library (== syside.Environment.get_default()) - this only preserves element IDs between serialization and deserialization. However, it is only used in the des.link step.

Without MembershipImports, all elements from the standard library with qualified names have deterministic element IDs as described in the specification. Next release extends it also to their owning_memberships which will cover most use cases as unnamed elements cannot be referenced textually anyway.

I still get empty diffs with

map = syside.IdMap()
for mutex in syside.Environment.get_default().documents:
    with mutex.lock() as dep:
        map.insert_or_assign(dep)

and loading both the JSON and Sexp from files from a previous run (both v0.7.2 and unreleased version).

Robert_Karban · September 19, 2025, 5:03pm

I think I’ve found the difference.
Following Simonas’ suggestion I use different options to create the json:
def _create_json_writer() → syside .JsonStringWriter:

json_options = syside.JsonStringOptions()

json_options.include_cross_ref_uris = False

json_options.indent = False

writer = syide .JsonStringWriter(json_options)

return writer

In particular,
json_options.include_cross_ref_uris = False
makes a difference.
it takes out lines such as:
{ “@id”: “f4b54269-d25a-4c93-af8e-b6065caf66aa”, “@uri”: “file:///.venv/Lib/site-packages/syside/sysml.library/Kernel%20Libraries/Kernel%20Semantic%20Library/Base.kerml” },

I cannot use a json that has those local URIs because I’m using to a v2 API compliant repository and therefore cannot have references to local files.
If I include them, then deserialize works though.

So how can I make that work without those local URIs?

Daumantas · September 22, 2025, 6:24am

This is a known limitation of the current implementation. This matches XMI exports where references are serialized as href="[<relative url>#]<element id>", otherwise references are opaque, and either the entire model has to be flattened into a single JSON (expect GiBs even for small-to-moderate non-trivial models), or the model can reference absolutely anything in the entire world.

Maybe we could add a callback to resolve unresolved local references for this case.

Robert_Karban · September 22, 2025, 6:40am

ok.
is there a work around so that serialize/deserialize become symmetric?
I am doing heavy post processing of the json to be able to deserialize but I don’t catch all the cases yet.

Daumantas · September 22, 2025, 7:32am

Currently, only merging all JSONs into one should work although I wouldn’t recommend it. I will look into relaxing initial link step into deferring unowned references without URIs to model.link(...) step.

Daumantas · September 22, 2025, 11:22am

Changed deserializer to defer unowned unresolved references, owned unresolved references will still emit an error because a model subtree is missing. There will be no additional changes needed as IdMap will handle this automatically, though with potentially worse performance, @uri will mainly be a performance and readability optimization.

This may be included in the next release.

Robert_Karban · September 22, 2025, 12:14pm

@Daumantas I’m not sure what that means for me.

There’s a new release or i have to change my code?

Daumantas · September 22, 2025, 12:18pm

Release should be out soon, no changes will be needed once its out. Until then, deserialization without @uri will fail.

Robert_Karban · September 24, 2025, 2:32am

Let me know when the release is out @Daumantas . Thank you

Daumantas · October 3, 2025, 11:33am

Hi @Robert_Karban, v0.8.0 has been released with the fix.

Robert_Karban · October 6, 2025, 12:12am

Thank you @Daumantas 0.8.0 works great!

Topic		Replies	Views
Json deserialization requires elements to be in order Bug Reports resolved , automator	16	106	October 8, 2025
Json serialization/deserialization bugs Bug Reports automator , resolved	6	85	November 18, 2025
SysIDE Pro 0.3.0 Changelogs	0	16	November 11, 2024
Syside crash when convert json to sysml textual Bug Reports automator	2	31	November 25, 2025
SysIDE Pro 0.5.0 Changelogs	0	17	February 11, 2025

Json serialization/deserialization is not symmetric

Related topics