Syside solver and use of jq or Powershell formatting does not match web examples

Syside Bug Reports
1

Hi,

Issues with the solver and jq noted. Please let me know if you can duplicate or if I’ve done something wrong. Thanks.

Not being a wizard with jq, but only following the webpage examples at Reading the solver’s output — Syside I get the following output for CMD, PS, and Bash shells. I ran all three because there have been inconsistencies between them for the solver command line previously.

The results are:

+====================+
| With Windows 10 PS |
+====================+

> $PSVersionTable.PSVersion

Major  Minor  Patch  PreReleaseLabel BuildLabel
-----  -----  -----  --------------- ----------
7      6      6

PS> syside solver check -i package_pump.sysml
{
  "result": {
    "subject": "Pump",
    "expression": "assertions[2]",
    "determination": "determined",
    "outcome": {
      "shape": "assertions",
      "assertions": [
        { "name": "Pump::flowOk", "status": "satisfied", "counterexample": null },
        {
          "name": "Pump::pressureOk",
          "status": "violated",
          "counterexample": {
            "values": [
              ["Pump::pump::pressure", "11"]
            ]
          }
        }
      ]
    },
    "causes": []
  },
  "warnings": [],
  "exit_code": 1
}

+=================================+
| Using native PS commands and jq |
+=================================+

PS> syside solver check -i package_pump.sysml | ConvertFrom-Json |
>>     ForEach-Object { $_.result.outcome.assertions } |
>>     Format-Table name, status
ConvertFrom-Json: Conversion from JSON failed with error: Unexpected character encountered while parsing value: ?. Path '', line 1, position 1.

+==============+
| and using jq |
+==============+
PS> jq --version
jq-1.8.2

PS> syside solver check -i package_Pump.sysml | jq -r '.result.outcome.assertions[] | "\(.status)\t\(.name)"'
jq: parse error: Invalid numeric literal at line 1, column 3

+==============================+
| With Windows 10 CMD using jq |
+==============================+

CMD> syside solver check -i package_Pump.sysml | jq -r '.result.outcome.assertions[] | "\(.status)\t\(.name)"'
The system cannot find the path specified.

+===============+
| With Git bash |
+===============+
$ echo $BASH_VERSION
5.2.37(1)-release

$ syside solver check -i package_Pump.sysml | jq -r '.result.outcome.assertions[] | "\(.status)\t\(.name)"'
jq: parse error: Invalid numeric literal at line 1, column 3

V/r,

@Richard_Wallace

2

You haven’t done anything wrong. On Windows, when syside’s output goes into a pipe or a file, it comes out as UTF-16 instead of UTF-8. jq and ConvertFrom-Json both expect UTF-8, so they choke on the first two bytes. That’s the ? in the PowerShell error and the “column 3” in the jq one. I’ve raised it internally.

Until that’s fixed, setting PYTHONUTF8=1 before running the solver makes the output plain UTF-8, and the examples from the docs then work as written.

PowerShell:

$env:PYTHONUTF8 = 1
syside solver check -i package_pump.sysml | jq -r '.result.outcome.assertions[] | "\(.status)\t\(.name)"'

CMD has a second, separate problem: it doesn’t treat single quotes as quotes, so the | inside the jq filter gets read as another pipe. That’s where “The system cannot find the path specified” comes from. The simplest way around it is to put the filter in a file, e.g. summary.jq:

.result.outcome.assertions[] | "\(.status)\t\(.name)"

and then:

set PYTHONUTF8=1
syside solver check -i package_pump.sysml | jq -r -f summary.jq

Thanks for running it in all three shells; that made this quick to pin down.

Adam Layne, PhD

3

I have tried using UTF-8 as the default encoding on Windows but Powershell butchers anything non-ASCII during its automatic conversion of text to UTF-16 (or more likely it converts any narrow and UTF-32 string to a wide string using the default code page), and I have not found a way to disable this behaviour. From my testing, neither changing the code page worked.

This is with forcing Python to output UTF-8 and piping the output, both to file and echo:

image

While redirecting UTF-8 output to a file with cmd.exe incorrectly sets UTF-16 file encoding anyway leading to garbage.

TLDR: there is no good way to use UTF-8 output with Python on Windows, there is nothing to fix. I am also not a fan of Windows being the only platform which mainly uses wide strings everywhere.

After further checking, this seems to be an issue with Python standard streams. Bypassing them and outputting unicode directly to C stdout/stderr works correctly even if the automatic widening conversion remains.

4

Thank you @Adam_Layne and @Daumantas for your analysis. :grinning_face:

The whole point of putting in this bug report is to say, “If you have an example on your web pages, make sure it works.” Any caveats, utilities, and environment settings should be mentioned on the web page. I don’t want to be a nitpicky user, but if you want people to use your products, make sure any examples on your public pages are correct. I know it’s difficult to anticipate all environments, so it’s best practice to state what the environment should support for the example to work correctly.:face_with_raised_eyebrow:

As you support both Windows and Unix derivatives, you’ve signed up for at least a two-string testing discipline.

I’m more than happy to be a beta tester, as it lets me teach/consult better, since I’ll know the “gotchas” of using the Syside, Sysand, and CLI tools (and recommended utilities). :wink:

So, to be useful, what I’ll do is “whack” away at this jq issue and write up my findings and post them back on this thread for your analysis.

V/r,

@Richard_Wallace

5

Howdy @Adam_layne, @Daumantas, and all the ships at sea! :grinning_face:

After some experimentation, I did get things to work correctly for CMD, PowerShell (v7), and Bash shells under Windows.

The solution is in two parts.

  • Part 1: convert the syside solver UTF-16LE file to UTF-8
  • Part 2: run with an input file to obviate the need for special shell treatment of jq formatting

Given the SysML v2 specification:

package Pump {
    private import ScalarValues::Integer;
    part pump[1] {
        attribute flowRate : Integer = 40;
        attribute pressure : Integer;
    }
    assert constraint flowOk { pump.flowRate <= 50 }
    assert constraint pressureOk { pump.pressure <= 10 }
}

And issuing the sysside solver command for all shells (CMD, PS(7), and Bash):

> syside solver check -i pump.sysml > solver.out

This results in a UTF-16LE encoded file solver.out due to the I/O “>” operator in all shells under Windows.

To convert the file for each shell, here are the tested steps. For those who want to pipe commands together or use I/O redirection, not all shells treat jq options the same. Hence, the jq options are put into a file.

I created the file with Windows Notepad.exe, which defaults to UTF-8. This is important for jq later!

.result.outcome.assertions[] | "\(.status)\t\(.name)"

Let’s call the file summary.jq as proposed by @Adam_Layne

Next, I’ll go through each shell and convert the UTF-16LE solver.out file to UTF-8.

CMD

Let’s check our code page with the chcp command. Starting with a newly created CMD shell, the native UTF-16 code page is 437 (OEM United States (Original MS-DOS console character set with line-drawing)

CMD> chcp
Active code page: 437

We now can change that to UTF-8 code page (for a list, see Windows code page - Wikipedia )

CMD> chcp 65001
Active code page: 65001

Because we have changed the code page, the I/O redirect > now produces UTF-8 output, and we can issue the command:

CMD> type solver.out > solver2.out

The solver2.out file is now in UTF-8 format.

PowerShell (v7)

For PS(v7), the command line is PFM ™!
The process is:

PS> Get-Content "solver.out" | Out-File "solver2.out"

The solver2.out file is now in UTF-8 format. Why? Here’s why:

  1. Get-Content reads the BOM, decodes the UTF-16LE bytes, and loads pure text into memory.
  2. The pipeline passes that pure text to the Out-File parameter.
  3. Out-File invokes PowerShell 7’s default encoding parameter (utf8NoBOM, i.e., raw UTF-8) and writes the text as clean, BOM-less UTF-8 bytes.

The solver2.out file is now in UTF-8 format.

Bash

For Bash, the conversion utility iconv is used:

iconv -f UTF-16LE -t UTF-8 solver.out > solver2.out

The solver2.out file is now in UTF-8 format.

===================================================================

In all shells, the summary.jq file has to be in UTF-8. Remember the summary.jq file? Yup, it has to be in UTF-8 to work.

jq -r -f summary.jq solver2.out
satisfied Pump::flowOk
violated Pump::pressureOk

QED

6

Of course the simplest correction for all of this is to have the tools write files in UTF-8 encoding, but there may be other issues with that simple fix.

V/r @Richard_Wallace

7

I too would like Windows to at least default to proper Unicode conversions but the code pages are not that. This is the same issue that requires programs to define wmain on Windows if they want to support Unicode inputs as wide strings do not go through narrowing conversion with some arbitrary code pages. Tried narrow main early on and some Unicode characters came in garbled up, instead we perform narrowing manually which preserves input characters.

See cli — Syside for more details. Windows is just a mess. (Fun fact: Windows also automatically inserts carriage returns to file output unless they have been opened in binary format)