We should not ignore errors or an empty SDK platform path since that means XCTest imports and test execution might silently fail on macOS with no indication to what the problem is.

This PR:

• Updates the SPMPackageEditor library so it compiles against ToT SwiftSyntax
• Removes spm-manifest-tool, instead conditionally compiling package editor functionality into the Commands module
• Refactors the library a bit to fix some bugs and make it more extensible
• adds support for adding products

Possible future directions (not in this PR):

• deleting products/targets/dependencies
• a swift package upgrade command that could update requirement specs
• Use the PackageSyntax infra to emit manifest-related diagnostics with source locations

ORIGINAL DESCRIPTION

SR-13566

This pull request changes the regex pattern on line 181 of swift-package-manager/Sources/PackageLoading/ToolsVersionLoader.swift to allow any combination of horizontal whitespace characters between "//" and "swift-tools-version" in the swift tools version specification in Package.swift

The old regex pattern is ^// swift-tools-version:(.*?)(?:;.*|$)

The new regex pattern is ^//\h*?swift-tools-version:(.*?)(?:;.*|$)

NEW DESCRIPTION (2020-11-04)

This PR proposes the following changes to the parsing of the Package manifest and its Swift tools version specification, starting from Swift Next (assumed to be Swift 5.4):

• Allow leading (any combination of all) whitespace characters in the Package manifest, instead of just line feed characters (U+000A).

  Rationale: This allows SwiftPM to skip all initial whitespace-only lines (if any) when looking for the Swift tools version specification, eliminating a common source of bug for Swift packages. Additionally, this allows SwiftPM to recognise all Unicode line terminators, because all of them are whitespace characters.

• Allow any combination of all horizontal whitespace characters¹ immediately after the comment marker (//) in the Swift tools version specification, instead of just a single space (U+0020).

  Rationale: Sometimes a user may prefer to follow the comment marker with a character tabulation (U+0009) or 2 space characters, or some other horizontal whitespace characters, in order to line up developer (//) and documentation (///) comments throughout the source files. For example:

  //  This is a demonstration of how comments may be lined up.
  /// A struct that represents nothing.
  struct Foo {}
  

• Allow any combination of all horizontal whitespace characters immediately before the version specifier in the Swift tools version specification.

  Rationale: This mirrors the most common style of type declaration in Swift sources. For example,

  let foo: Foo
  

• Recognise all Unicode line terminators instead of just line feed (U+000A) in the Package manifest up to the line terminator that immediately follows the Swift tools version specification line².

  Rationale: With Swift becoming supported on more and more platforms (many of them Unix-unlike or POSIX-incompliant), SwiftPM will see more and more Package manifests that use line terminators other than line feed. Further, even though many systems are not yet supported, Swift source files can still be edited on them, and many IDEs/editors silently changes the line terminators to the platforms' native ones when saving the edited files. SwiftPM needs to recognise all possible line terminators³ before it becomes a problem

• End the silent fallback to using Swift 3.1 as the lowest supported version. This used to happen when the Swift tools version specification is malformed AND SwiftPM couldn't find either swift-tool or tool-version in the specification.

  Rationale: It is actively harmful to hide this behaviour from the user. It is also actively harmful to decide on a Swift version without user consent. With source breaks from past Swift releases accumulated, this could result in unexpected behaviour in Swift packages. This behaviour was originally intended to peacefully accept Swift packages created before the Swift tools version specification was required. However, this purpose has become outdated.

Additional user-facing changes as the result of implementing the changes above:

• If the manifest has a formatting error, SwiftPM now identifies more accurately where the error is, and provides to the user a more detailed description/explanation and a suggestion tailored for each error.

• SwiftPM now throws an error if the manifest is not properly encoded in UTF-8.

Additional SwiftPM client-facing changes as the result of implementing the changes above:

• ToolsVersionLoader.Error.malformedToolsVersion(specifier: String, currentToolsVersion: ToolsVersion) is replaced by more granular error cases.

• ToolsVersionLoader.split(_ bytes: ByteString) -> (versionSpecifier: String?, rest: [UInt8]) and ToolsVersionLoader.regex are replaced by the Substring-oriented ToolsVersionLoader.split(_ manifest: String) -> ManifestComponents.

¹ Whitespace characters consists of horizontal and vertical whitespace characters. No whitespace character can be both horizontal and vertical. All vertical whitespace characters are line terminators, and vice versa. In the PR's implementation, horizontal whitespace characters are not identified directly, but through the process of elimination: All line terminators are "stripped" from the manifest first, then the horizontal whitespace characters are identified by being whitespace characters.

² Everything after the Swift tools version specification, such as the initilisation of the Package instance, is handled by the Swift compiler's lexer. It's beyond the power of ToolsVersionLoader and SwiftPM's manifest-loading in general.

³ Only Unicode line terminators are recognised, so line terminators in some other encodings such as EBCDIC are not recognised. This is fine because if a Package manifest isn't encoded in UTF-8, then it's already entirely unreadable by SwiftPM.

Initial implementation of https://forums.swift.org/t/pitch-package-manager-command-plugins/53172.

This feature is experimental until approved, so it requires a tools version of 999.0 in the package that declares the command plugin, and requires SWIFTPM_ENABLE_COMMAND_PLUGINS to be set to 1 in the environment when invoked.

This implementation temporarily uses a swift package plugin <verb> form for invoking command plugins. Per the pitch and proposal, plugin-defined commands are intended to be invocable using the shortcut swift package <verb> when the feature is complete (there is discussion of even supporting swift <verb> but that is not part of the current proposal).

Motivation:

Implementation of a new feature being pitched, kept under a feature flag until the proposal is accepted.

Staging:

This feature is being developed incrementally across this PR and follow-on PRs. This PR it is in a state where it could be merged to get most of the functionality into the mainline (behind the feature flag), including all the planned changes to PackageDescription and PackagePlugin. Remaining items of the full feature would then be landed in future PRs. The SWIFTPM_ENABLE_COMMAND_PLUGINS guard flag will only be removed once the evolution proposal has been approved and the feature has been fully implemented including any amendments.

Modifications:

• add the new enum cases for the command plugin capability to PackageDescription
• add serialization of those cases in PackageDescriptionSerializer
• deserialize them in the ManifestJSONLoader and add to the package model
• add manifest source generation for the command capability
• add ability to describe packages that have command plugins
• add a unit test to check loading a manifest with a command capability
• add protocol declarations for command plugin entry point in the plugin API
• ability to invoke the command plugin in libSwiftPM and unit tests
• add a unit test to check round trip calling into the plugin
• add ability to invoke the command plugin from CLI
• made the host-to-plugin communication be bidirectional and interactive
• ability for plugin to call back into SwiftPM for services
• ability to create symbol graphs for a target when a plugin requests it
• ability to run a build when the plugin requests it

Remaining in upcoming PRs:

• request approval to run command plugins that need special permissions
• implement the option to list the available command plugins
• support qualifiers for ambiguous plugin commands
• ability to run a test when the plugin requests it
• build any executables needed by command plugins before invoking the plugin
• provide access to toolchain executables in the map sent to plugins
• add async to all the plugin APIs (in separate PR, depends on back deployment)

I reworked my approach from PR #2835 to not only operate on GitRepository but on any kind of repository from within Repository Manager.

SR-12851

This PR requires the Lock Improvements in swift-tools-support-core to be merged.

The bootstrap script needs to build the PackageDescription and PackagePlugin runtime support libraries universal when cross-compiling. This PR builds on https://github.com/apple/swift-package-manager/pull/3464, so it will be easier to review that PR first.

Motivation

Even when invoking the bootstrap script with flags to build universal for macOS x86_64 and arm64, the PackageDescription and PackagePlugin libraries ended up single-architecture. This is because the bootstrap script copied them from the CMake-built artifacts instead of the swiftpm-built artifacts.

PR https://github.com/apple/swift-package-manager/pull/3464 resolves the complication of PackageDescription having both a 4.0 and a 4.2 variant.

Modifications:

• in the bootstrap script, build SwiftPM so that Swift interface files are emitted
• consolidate some of the bootstrap script logic for installing dynamic libraries and their companion modules

rdar://75186958

Motivation:

This enables cross-compiling SPM for other hosts, such as Android.

Modifications:

In addition to --cross-compile-hosts, add a new bootstrap flag, --cross-compile-config, that will pass in a JSON file with the destination cross-compilation flags.

Result:

SPM can be cross-compiled for another OS, with Android as the first example.

This change introduces a build verification step that attempts to detect scenarios where a target contains an import of another target in the package without declaring the imported target as a dependency in the manifest. This is done via SwiftDriver's import-prescan capability which relies on libSwiftScan to quickly parse a target's sources and identify all imported modules.

Work in Progress This is initial commit for SR-473

We need to generate version data for every package on build. Now it creates a new folder versionData inside .build dir and creates a swift file for every package.

Could you please provide any feedback for the initial implementation.

• Should these files be inside .build dir or somewhere else?
• Should we generate a new .swift file for every Package or place all data in 1 file?
• Any other suggestion?

ToDo:

• [x] Add unit test
• [x] Add better error handling
• [x] Merge PR #225
• [x] add additional information to Root package (local changes)

| | | |------------------|-----------------| |Previous ID | SR-40 | |Radar | None | |Original Reporter | @gribozavr | |Type | New Feature | |Status | Resolved | |Resolution | Done |

Attachment: Download

Sub-Tasks:

• SR-387 Undef. symbols when linking SwiftPM
• SR-359 Undef. symbols when linking to Foundation (ARM)

Issue Description:

It would be good to get Swift compiler running on Raspberry Pi, as well as having a cross-compiler from x86 to Raspberry Pi.

This pull request should complete the implementation of SE‐0226: Package Manager Target Based Dependency Resolution. @hartbit already taught the resolver to only care about products in #2424. This takes that work further so that the resolver only cares about products that are actually used.

This pull request is sizeable, so I’ll elaborate the main points here.

The changes broadly fall into four groups:

1. The manifest is now able to compute the downward‐facing products required for any upward‐facing product set. This is mostly housed in PackageModel/Manifest.swift, and boils down to turning @hartbit’s allRequiredDependencies into dependenciesRequired(for:), along with the input type ProductFilter.

2. The resolver now operates on abstract nodes. See DependencyResolutionNode for the nitty‐gritty details. Essentially, it resolves individual products instead of whole packages. The new node type has simply been slotted in to replace the PackageReferences that were in use before. None of @kiliankoe’s Pubgrub decision making processes has been changed in any way. The diagnostics have also been left alone except for the inclusion of the specific product alongside the package name where relevant.

3. Dozens of new method parameters and instance properties have been added to various types and methods in order to pass the new information along from where it originates to where it is needed. Unfortunately, this affects a whole swath of things across the PackageModel, PackageLoading, PackageGraph, Workspace, Commands and SPMTestSupport modules, but all of these changes are straightforward.

4. Many tests required fixing. Most of them merely needed to provide the new parameters required in order to satisfy the new API. Several were expecting failures that were no longer reachable, and needed dependencies to be made explicit so that the resolver would even enter the code branch in question. All but three are straightforward. (See below.)

While I was working through the tests, three things came up which need a closer look. They are tests that expected behaviour at least partially at odds with SE‐0226.

1. The test suite expected unreachable executable products to be available. swift run dependency‐executable is apparently supposed to work, even if no dependency is declared on that product. I can see how that could be useful, and it wouldn’t surprise me if users are taking advantage of it in their workflows. But SE‐0226 says this:

   Another example of packages requiring additional dependencies is for sample code targets. A library package may want to create an executable target which demonstrates some functionality of the library. This executable may require other dependencies such as a command-line argument parser.

   So including such executables would undermine one of the motivating use cases for SE‐0226. As such, what I have implemented right now does not resolve such executables by default.

   However, as a compromise to maintain backwards compatibility, I have channeled the argument provided to any swift run invocation through to the resolver, so that if you explicitly ask to run an executable of an immediate dependency, you will still be able to. Since it has some minor drawbacks, I consider this an interim solution. This implementation will suffice for now, but a separate evolution proposal should probably be made to design something better if we want to continue supporting this long‐term. See the Fix‐Me in the initializer for PackageGraphRoot for more details.

2. The test suite expected unreachable targets to be available to a swift build --target dependency‐target invocation. Since there is no infrastructure in the resolver to handle targets, there is no easy way to locate or load such a target. Since it seems backwards to SE‐0226 and has no practical use I can think of, I simply removed the test.

3. The test suite expected legacy system packages to work without an explicit target dependency declaration. Such packages have been triggering a deprecation warning since Swift 4.2. Continuing to support them implicitly is problematic, because there is no way to know if that’s what a package is until after fetching and loading it, so it would defeat the whole of SE‐0226.

   As a compromise to maintain as much backwards compatibility as possible, I have taught the manifest loader to internally convert such a package into a modern‐style one with a system target and product, which inherit the package’s name and details. This allows the current toolchain to still use such dependencies by referencing the synthesized product. However, if the client package wants to support both old and new toolchains at once (such as 5.1 and 5.4), it will have to make use of #if compiler, because the old toolchain will be unable to locate the declared product, whereas the new toolchain will not resolve an undeclared product.

Add CMake based build support for the missing components as the Windows build depends on this still. This allows us to work towards enabling the swift package collection and registry functionality.

This PR change allows the generation of targets containing only header files.

Motivation:

Closes https://github.com/apple/swift-package-manager/issues/4806 . Currently, if we attempt to create a target that does not contain source files, the Swift Package Manager returns an error and prompts us to put the source files. Therefore, it is not possible to create a CLangTarget that only contains header files.

Modifications:

Changed so that if a header file exists, the Target generation process continues even if no source file exists.

Result:

With the changes in this PR, a target that does not contain source files but does contain headers is a CLangTarget.

Description

Make a tool aan print the xcodetarget dependencies. This is always an empty list.

Expected behavior

To list the dependencies as the doc describes

Actual behavior

empty list

Steps to reproduce

see description

Swift Package Manager version/commit hash

with xcode 14

Swift & OS version (output of swift --version && uname -a)

swift-driver version: 1.62.15 Apple Swift version 5.7.1 (swiftlang-5.7.1.135.3 clang-1400.0.29.51) Target: arm64-apple-macosx13.0 Darwin MacBook-Pro-3.lan 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T6000 arm64

Description

This relates to discussion about AirbnbSwift Where it was confirmed that the swift package index does not detect incompatible libraries because they build per platform, but as described the build does not fail.

The workaround for now seams to be to add an error the Package.swift, which seams not the way to go :).

Expected behavior

Do not build of binary target does not build for the platform you are building for. This can easily be derived from the json of the binary.

Actual behavior

It just builds but crashes when run before an executable actually starts.

Steps to reproduce

See discussion https://github.com/airbnb/swift/discussions/197 & https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/issues/2116

Swift Package Manager version/commit hash

the one that ships with xcode 14 and above

Swift & OS version (output of swift --version && uname -a)

swift-driver version: 1.62.15 Apple Swift version 5.7.1 (swiftlang-5.7.1.135.3 clang-1400.0.29.51) Target: arm64-apple-macosx13.0 Darwin MacBook-Pro-3.lan 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T6000 arm64