The Swift Package Index is the place to find Swift packages!

I've been working on my docs for https://swiftpackageindex.com/ChimeHQ/ChimeKit. I was surprised to see the sidebar was empty, showing just the target name and "No data available."

In Xcode's doc viewer, I see this:

I'm pretty new to docc, so I could definitely be doing something wrong. I also cannot quite figure out how to generate local static web content, but I'm working on it...

I wonder whether it would be possible (good?) to display whether or not a package exclusively uses a binaryTarget or not?

What I mean by this is that there are some Swift packages (example) who's sole purpose are to (re)distribute and make available precompiled xcframeworks to SPM. In these scenarios, there is no source code.

Sven and I agreed that we should start a slightly more structured issue around the proposal of a metadata file that packages may add. So here goes.

Why?

The primary goal of this site is to help people make better decisions about the packages they are choosing. The metadata we currently use to help people make those decisions comes from the manifest, the repository, and GitHub. There's more we could do with better information though, so we're considering proposing a standard metadata file that package authors can use to inform the Swift Package Index better. It would also mean that other hosting providers (for example GitLab and Bitbucket, as well as self-hosted repositories) would be on equal footing with GitHub.

What is the file, and where is it stored?

We see this metadata file hosted in the root of a package repository. It's better for everyone if the package author is in control of the file, and of course, it means that other projects can also take advantage of the information.

What information would it include?

This thread aims to gather feedback from the community on what information would be useful in this metadata file. I will update the list here as feedback comes in. Here's what we have so far:

• Package information:
  • Abstract (thanks Erica)
    • A shorter, one line description.
  • Description (details here)
  • Categories/Tags:
    • Manually defined (thanks to Erica)
    • Some tags could potentially be derived from imported frameworks (see here). This would not be in the metadata file of course, but I think is worth mentioning here.
  • Home page URL
  • Documentation page URL
  • Example project URL (thanks James)
  • Auxiliary URLs (thanks Dave)
    • A set of any number of other URLs to other resources, we’d need to capture both a URL and the text to use as the link.
  • Is the package deprecated?
    • If so, is there a successor package?
  • Related packages: (multiple items of) (thanks James)
    • Package URL (The name and all other metadata can be derived from this)
  • License (thanks Johan) - See comments here too.
  • Maintainer: (thanks Erica) (exactly one of)
    • Email address
• Author information: (multiple sets of)
  • Name
  • Email address
  • Personal URL (Home page, Twitter, GitHub page, etc…)
• Funding/sponsorship/donation information: (thanks Max)
  • Does the package accept funding?
  • Funding URL
• Other platform support status:
  • Linux (thanks Max)
  • A boolean is the simplest way to declare support.
  • Explicit support for named versions of Linux is more comprehensive.
  • Windows, and other platforms as they are added. (thanks Erica)

By definition, as the file will not exist in all repositories, all of this data will be optional. No package author should be required to add any data that they are not comfortable with sharing.

There is a valid issue with versioning the metadata file brought up by Mattt here. For the Swift Package Index, while we would potentially store this information against the versions, The Swift Package Index would use the latest version of the file on the default branch to build the package pages from.

Structured or Unstructured data

There's an argument to make all non-technical metadata a "tag" like structure. For example indicating categories, Linux support, and other things with string tags as mentioned by Erica here.

We want to bring as much of the information that's needed to judge the quality of a package into one place. For example, instead of having to check how many pull requests/issues there are and when the last one was closed, we bring that in automatically, right alongside information about what versions of Swift the package supports, and whether the stable release is the right one to target, or if there's actually a beta which would better suit your needs.

All of that data so far is structured as it comes from the manifest, from GitHub, and from the repository itself. There's a place for unstructured/tag-based data, but I don't think it completely replaces the need for structure.

We also want to use some of this structured data to drive a "quality score" for a package. I don't think it's clear yet whether this quality score is made public, or just used internally for search ranking (we have a version of this already) there are pros and cons to both. But, if metadata is just tag-based, it's much harder to do that. Especially when tags can be typed incorrectly or interpreted in different ways (do linux and ubuntu-18.04 get points for supporting Linux, where ubuntu1804 doesn't?). It's definitely a trade-off. -- Just a note, I'm not saying packages would definitely get an increased score for supporting Linux, it's just an illustration.

Scope of this thread

I think it's worth keeping the discussion to the information at this point, rather than the specifics of the format that we’ll use to represent it. That's a separate discussion, and the data we decide to include will influence the format.

At this point, we should include ALL suggestions for metadata. Of course, it's fine to put forward views on why you feel a piece of metadata shouldn't be considered. But I won't remove any until we've got a comprehensive collection of everything under consideration though. I’ll keep this list above up to date as more suggestions are added in comments below.

Eventually moving metadata into Package.swift

If this process is successful, it's worth considering whether this metadata should be merged into the Package.swift manifest through the evolution process. It's an idea, but probably only for some of the metadata.

The package manifest holds information about the technical details of the package, and I think we should be careful mixing in descriptive metadata in with that. So, if we see that Linux support is something we see people use this metadata file for. We think that would make a great addition to the official manifest. However, for things like description, tags, author information, etc… a separate file feels better.

We've briefly discussed this in the context of the ReleaseNotes tool: what if we re-directed non-existing packages to Github URLs.

This is just throwing this together quickly to try and discuss.

The reason I'm picking this up again is in the context of package dependencies:

Package URL resolution

One issue we'll need to address is how to resolve URLs we find in Package.resolved and the "product dependency tree". We could look up each one against the index and replace it with the package id if we have a match and keep the URL only if we don't.

This is going to cause quite a number of lookups for some of the packages. (To do: Double check the actual impact.)

That raises the question how we're actually going to use these links to packages. Presumably, we'll want to allow links to package pages. So how about we simply either convert or re-interpret https://github.com/foo/bar.git as https://swiftpackageindex.com/foo/bar and add a redirect handler to the original URL if we don't have the package?

If we do this we don't really need to look up any dependency URLs at all.

There's one obvious downside with this: it assumes that any URL we don't host (at the package level) is a github.com and not any other source code hosting provider. (Unless of course we try them subsequently in priority order...)

However, if all we ever do is link to package pages from dependency URLs then substituting them with package IDs is actually detrimental. We'd only be doing a lookup for a package ID and then look it up again to convert it to a URL when following a dependency.

Unless of course we don't convert them to package IDs but URLs. But then why do a look up in the first place? If all we want is the URL, we might as well optimistically guess it.

In the context of the ReleasNotes tool we could leverage the same mechanism: substitute SwiftPackageIndex.com for all GitHub.com url without checking the server first. While our release notes aren't as fully functional as Github's yet, they're also not a mile away either. And you can click through to Github's release note from each of them.

The biggest problem with substituting SPI urls was the handling of 404s, which this would solve.

I searched for “toml” on the iPad, tapped he share icon and selected “Copy link”. The resulting url was

https://swiftpackageindex.com/search

i.e. it dropped the search term and therefore wouldn’t link to the actual results.

It might feel obvious that this file should be in a JSON format. However, this file is going to be written entirely by humans and read by both humans and computers. The ability for a human to accurately read and input this data is important.

Another important feature of the format we choose is how it handles multiline strings. The summary/description fields will be multiline strings.

Finally, comments are also important. Package maintainers will probably be starting with a template file and we can provide guidance in comments.

YML

• Pros
  • Easy for a human to read and write.
  • Easy for a computer to read.
  • Good support for multiline strings.
  • Good support for comments.
• Cons
  • Requires a dependency for any Swift app that wants to read the metadata. For example, Yams or Yaml. (thanks Max)
  • Quirks around interpreting some data formats (thanks Sven and Max)

TOML

• Pros
  • Easy for a human to read and write.
  • Easy for a computer to read.
  • Good support for multiline strings.
  • Good support for comments.
• Cons
  • Requires a dependency for any Swift app that wants to read the metadata. For example, TOMLDecoder or Toml. (thanks Max)

JSON

• Pros
  • Trivial for computers to read.
• Cons
  • Harder for a human to read and write.
  • Multiline strings are awkward.
  • No support for comments.

Swift (via PackageConfig or similar)

• Pros
  • Easy for a human to read and write.
  • Trivial for computers to read.
  • Good support for multiline strings.
  • Good support for comments.
• Cons
  • Every package would need a dependency on PackageConfig. To be clear, this is very different from requiring a dependency for reading the metadata, this is a true package dependency. For example, Alamofire would become dependent on PackageConfig if they wanted to implement this metadata. (thanks Gal)

Property Lists (thanks James)

• Pros
  • Easy for a human to read and write using Xcode.
  • Trivial for computers to read. (No third party libraries as we can use Codable)
  • Good support for multiline strings.
• Cons
  • Very Apple platform centric. (thanks Max)
  • No support for comments.
  • Not super easy to read/write as a source file (XML)

This PR addresses the discussion and the todo items in https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/discussions/1661

More specifically:

• adds product type to the search view

• adds a new filter token for product type "type:plugin", "type:executable", etc (1 at a time, no inversion)

• adds the type search filter to the search, including visualization of the predicate

• adds plugin information to the package page

To Do:

• [x] icon for plugins. I'd really like this one https://fontawesome.com/v4/icon/plug but I don't know how to generate the image data for the css file. I've added a placeholder
• [x] this is my first time using vapor + fluentkit + postgres, I just picked up code that was laying around and stitched together this PR. Someone should have a look and see if it makes sense
• [x] I left a TODO in QueryPerformanceTests because I didn't know what number to put in for the new performance test
• [x] I (@finestructure) want to add a new baseline search view refresh query performance test to main before we merge this to get a feel for the impact of the additional join with products

I've run our test suite a lot in a docker container the last couple of days and found at least two different ways in which it can fail:

• connectTimeout

  <EXPR>:0: error: SearchTests.test_keywordSearchFilter : threw error "connectTimeout(NIOCore.TimeAmount(nanoseconds: 10000000000))"
		Test Case 'SearchTests.test_keywordSearchFilter' failed (10.047 seconds)

• Vapor/Application.swift:164: Fatal error: Application.shutdown() was not called before Application deinitialized.

I suspect they are also the cause for the benchmark test failures when running the test suite more than ~8-10 times on @daveverwer's machine.

I've observed these failures going back as far as 2.58.9. I.e. they're not related to any recent changes in PostgresNIO. They likely also go back further than that - it's just where I stopped looking for now.

I can typically hit them by running the test suite in a remote container in VS Code 1-3 times. By the third run I've hit this problem.

It seems (although this is just anecdotal) that running the suite in quick succession makes this happen more "reliably". This would explain why we're not seeing it in CI.

I suspect it's also a race condition problem that manifests itself on fast machines more easily. For instance, we've never (to my knowledge) seen these issues in CI, whereas I've managed to trigger it on the first run in a container. So given how often we run the test suite in CI, we surely should have hit these issues.

The reason I've now started to see this behaviour is because I've been trying to diagnose CI test failures on the redirect-non-existing-packages branch that happen on CI but not locally on macOS *). To that end I've been running the test suite in a container locally a lot and found these issues.

Unfortunately, this makes diagnosing the issues pretty hard, because I believe there are multiple issues at play.

*) This may be a third and unrelated issue, or it may simply be that somehow on that branch we're managing to make the flakiness of the two kinds of test failures listed above reproducible in CI for once. That's one of the things I'm trying to find out right now.

The last smoke test picked up a few 404s:

• https://swiftpackageindex.com/BenEmdon/CenteredCollectionView (still 404)
• https://swiftpackageindex.com/GEOSwift/GEOSwift (now 200)
• https://swiftpackageindex.com/Moya/Moya (now 200)
• https://swiftpackageindex.com/ashleymills/Reachability.swift (now 200)
• https://swiftpackageindex.com/llvm-swift/LLVMSwift (now 200)
• https://swiftpackageindex.com/pointfreeco/swift-tagged (now 200)
• https://swiftpackageindex.com/square/Valet (now 200)
• https://swiftpackageindex.com/stephencelis/SQLite.swift (now 200)

No specifics yet as this is a complex feature, but this is here to track progress towards this goal.

Any feature requests specific to the display of dependency information against a package would be great to add below.

Discussed in https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/discussions/2120

I was sure we had an issue open for having documentation pages missing from the Google index, but I can't find it.

After a rather confusing investigation with Google Search Console, I believe our missing documentation pages are being skipped during indexing or excluded from the Google index because we have multiple versions of every documentation set hosted and available.

We should set a canonical URL on every documentation page and point to the latest stable/beta/branch release using the same logic as we have for where the “Documentation” button links to.

We're currently showing the following info:

However,

.package(url: "https://github.com/soto-project/soto-s3-file-transfer.git", from: "1.1.0")

is only part of what's needed for the package manifest.

In your target, you also need to specify

.product(name: "SotoS3FileTransfer", package: "soto-s3-file-transfer"),

to use the dependency.

We actually have all the information to provide these bits as well, per product. I.e. we could generate a .product(name: "\(product)", package: "\(package)") line per product (typically just one).

We can cap this at some number to avoid an overflow for packages with lots of products.

We would not need to do this per significant version, it could be a common section below. (Yes, theoretically the default branch could have different products defined than the stable release but I don't think we need to take that sort of nuance into account.)

The fallback order should be:

• Latest stable release (Highest priority)
• Latest pre-release tagged version
• Default branch (Lowest priority)

I believe it currently only does:

• Latest stable release (Highest priority)
• Default branch (Lowest priority)

  I've put up a .spi.yml for both `ordo-one/package-benchmark` and `ordo-one/package-frostflake` that should try it out:

E.g.:

version: 1
builder:
  configs:
  - documentation_targets: [Frostflake]
    platform: linux

Originally posted by @hassila in https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/discussions/2184#discussioncomment-4386990

We've seen via #2179 that uploading large doc sets is problematic. The 1GB/80k files doc set takes 6mins to transfer via scp between MacStadium machines.

Even if there was a network issue on the day, we'll be hard pressed to upload doc sets of that size within the 10min timeout we impose on builds (and that includes the build and doc generation).

We should probably consider sending the zipped doc set to an intermediary which then deals with the upload asynchronously and with a more generous timeout and additional retries than we afford the current upload.

Not sure what the file size limits are for lambda but I could see that being a decent solution: send a zip to a lambda and let it deal with the S3 population.

Discussed in https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/discussions/2177

^{Originally posted by finestructure December 6, 2022} I feel like we've discussed this before somewhere but couldn't find any issues or discussions.

In a recent PR, someone commented:

it takes you to the Instrumentation documentation instead of Tracing without any UI to switch to the Tracing documentation

I actually initially overlooked this comment and then while helping with the PR ended up struggling to figure out if my fix worked, because I couldn't find the target picker 🙈

So I think there's something there that may need addressing.

It may be as simple as adding a drop-down chevron to indicate the breadcrumb is a menu here:

(and to the version segment as well while we're at it).