Usages from local filesystem indexes

Sysand How-to Guides
1

Continuing on my quest to figure out how to locally/ privately host sysand packages.

First attempt was via git repos, here: Sysand git repositories as usage sources

Now attempting with a local index per the docs Self-hosting a project index - Sysand User Guide

Seemingly following the docs to the letter and attempting to add a usage from the index hosted with a local webserver, sysand add urn:kpar:mymodel --index http://localhost:8080 reports an error error: incomplete projectEditableProject { inner: LocalSrcProject { project_path: “.” }, nominal_path: “.”, include_original_sources: false }.

Alternatively, trying to reference the index through the filesystem with a file url: sysand add urn:kpar:mymodel --index file:///path/to/sysand_index/sysand_envresults in the same error.

Similar to my problems trying to import sysand packages from git repositories, it isn’t clear to me what is incomplete about my project, or what next steps I should take to fix the problem.

On the topic of unclear errors, it also took me a minute to realise i needed to use the file:// prefix to reference an index on the local filesystem. Before using that I was just passing an absolute path and received error: relative URL without a base.I guess this is clear feedback to developers, but it wasn’t immediately clear that my error was not specifying an appropriate (IRI) scheme.

Thanks for the support!

2

Indexes can only be used via HTTP(S), so file:// URLs won’t work for this.

For me the following steps work:

  1. Go to any directory. If in a subdirectory of an existing project, sysand_env will be created at the root of the project, otherwise it will be in current dir. Run:
    sysand env
  2. Install any packakge to the sysand_env, with --path pointing to either a .kpar file or a directory with .project.json + .meta.json:
    sysand env install urn:kpar:mymodel --path /path/to/mymodel
  3. Start an HTTP server of sysand_env, which is either at the current directory or at the root of the project:
    python3 -m http.server -d sysand_env 8080

Then in another project’s directory:
sysand add urn:kpar:mymodel --index http://localhost:8080

I guess this is clear feedback to developers, but it wasn’t immediately clear that my error was not specifying an appropriate (IRI) scheme.

Error reporting will be improved. It hasn’t yet received much attention.

3

Hi Andrius,

Indexes can only be used via HTTP(S), so file:// URLs won’t work for this.

Good to know, thanks. Perhaps a note at the top of Self-hosting a project index - Sysand User Guide and Project metadata - Sysand User Guide . In the latter, the final paragraph “Projects in HTTP(S)…” can be interpreted to imply that it is possible to have indexes also without http(s) interfaces.

For me the following steps work:

Your steps here are almost exactly the steps I executed, giving the result in my post. The only difference is that I installed to the local index with sysand env install urn:kpar:mymodel --no-deps --path /path/to/mymodel.

Retrying following your steps explicitly, installing to the index referencing a .kpar gives the same error: incomplete project. I have no sysand.toml or environment variables that could be overriding --index.

I also note that the python http server stdout does not report any http get requests when i try to add the dependency in the new project. http server does report http get requests when requesting from a browser or with curl.

What exactly does error: incomplete project mean? The kpar I am trying to add to the index has had project metadata added per Publishing a package to Sysand Index - Sysand User Guide.

4

Hi Andrius,

Following your suggestion at Support workspaces · Issue #101 · sensmetry/sysand · GitHub

First test - direct https link to a kpar in github release

sysand add https://github.com/sensmetry/smart-home-hub-example/releases/download/v0.1.0/smart_home_hub_example-0.1.0.kpar works.

second test - direct https link to a kpar in github private index

(Yes, i know this repo was set up to be used with sysand add urn:… -–index https://…., but here I am testing the direct https method).

sysand add https://github.com/sensmetry/sysand-private-index/blob/main/packages/test_sysand_package/test_sysand_package-0.0.1.kpar
Result:

error: Failed to satisfy usage constraints:requested project(s) alternative nr 0 depends on sysand-private-index/packages/test_sysand_package/test_sysand_package-0.0.1.kpar at main · sensmetry/sysand-private-index · GitHub .

I guess that test package has internal dependencies that cannot be resolved.

This example also raised something new for me:

  • In my test directory, I had succesfully added smart_home_hub_example-0.1.0.kpar.
  • Cleaned the test directory, deleting sysand_env, meta.json and project.json. Forgot to delete sysand-lock.toml.
  • Tested addingtest_sysand_package-0.0.1.kpar above. Failed with error above.
  • Explored the results of sysand env list, sysand info, sysand sync.
    • Discovered sysand sync installed smart_home_hub_example-0.1.0.kpar.
    • This is correct, I had forgotten to delete sysand-lock.toml, which was still the lock file from my last test.
    • Unexpected: I had assumed that upon sysand add xxx, the lock file would be updated. It turns out sysand-lock.toml is only updated upon a successful sysand add.

Per the docs:

By default this will also update the lockfile and sync the local environment

I now assume it only updates the lockfile if sysand add was successful. In my example above, according to the docs, I would expect the lockfile to update to list even the faulty usage, so that standalone calls to sysand sync report the error on the faulty usage. This is probably the desirable behaviour - want the lock file kept up to date, even if faulty.

Third test - direct https link to a kpar in a remote git repo (gitea)

  • Attempt: sysand add https://…
    • Result: Same result as test 2 - error: Failed to satisfy usage constraints:requested project(s) alternative nr 0 depends on....
  • Attempt with —no-sync: sysand add https://… —no-sync
    • Result: same error
    • Expectation: with –no-sync, dependencies should not be traversed?

Sysand will be very useful. Some improvements in the error reporting to figure out root causes and next steps will be great!

Addendum:

I was just writing a feature request for something similar to pip show or pip index versions, then found you have sysand info -a IRI. This is very nice. It allows me to check if an IRI is resolvable without first having to attempt sysand add, then remove failures with sysand remove. Would have been a better way to perform the above tests.

Sysand git repositories as usage sources
5

This URL is for the web interface (i.e. the HTML page). You have to point directly to the kpar file by using the “raw” GitHub URL (right click on “Raw” button → copy link): https://github.com/sensmetry/sysand-private-index/raw/refs/heads/main/packages/test_sysand_package/test_sysand_package-0.0.1.kpar

6

you have to point directly to the kpar file by using the “raw” GitHub URL (right click on “Raw” button → copy link):

:heart: that solved test 2.

7

The intended behaviour is to update the lockfile and .project.json only if sysand add was successful. Right now the problem is that .project.json is updated unconditionally before trying to resolve the usage, which then results in any subsequent sysand sync or any command internally invoking package sync failing, unless the bad usage is removed. As it is undesirable to get into such a corrupt state, we have a fix on the way that only updates .project.json if the sync is successful. Lockfile itself will only be updated if the sync was successful.

Regarding links, same note as above applies – links to raw files instead of HTML pages must be used.

Regarding --no-sync, it does not download the projects (technically it does, to discover their usages and generate a checksum used in the lockfile, but these projects aren’t saved anywhere), but still generates the lockfile. Since the lockfile records the precise versions of each project in the dependency/usage (sysand documentation in general refers to direct usages of a project as usages, and all usages, including transitive ones as dependencies) graph, that means resolving (i.e. finding) each project in that graph, which in turn means all projects must be resolvable. To skip dependency resolution altogether, use --no-lock, then the usage will only be added to .project.json without checking anything, as --no-lock implies --no-sync.

1 Like
8

By the way, by passing -v parameter together with any command, debug logs will be printed. They include more information which may be useful for troubleshooting.

1 Like
9

Thanks Andrius. -v helped find the error for test 3 on my side. sysand doesn’t trust the certificate from my self-hosted gitea instance.