I upgraded to Swift 3.1 Then I updated to swift-protobuf 0.9.29 Compiled proto-gen-swift from the 0.9.29 sources and use the 0.9.29 version as a dependency.

Some protobuf files that were previously working correctly now generate hundreds of errors like:

/.../Work/Swift/Systems/****/.build/checkouts/***Protobuf-5523697989699535816/Sources/***Messages.pb.swift:1292:22: note: overloads for '!=' exist with these partially matching parameter lists: (Any.Type?, Any.Type?), (UInt8, UInt8), (Int8, Int8), (UInt16, UInt16), (Int16, Int16), (UInt32, UInt32), (Int32, Int32), (UInt64, UInt64), (Int64, Int64), (UInt, UInt), (Int, Int), (ContiguousArray<Element>, ContiguousArray<Element>), (ArraySlice<Element>, ArraySlice<Element>), (T?, T?), (T?, _OptionalNilComparisonType), (_OptionalNilComparisonType, T?), ((A, B), (A, B)), ((A, B, C), (A, B, C)), ((A, B, C, D), (A, B, C, D)), ((A, B, C, D, E), (A, B, C, D, E)), ((A, B, C, D, E, F), (A, B, C, D, E, F)), (LazyFilterIndex<Base>, LazyFilterIndex<Base>), ([Key : Value], [Key : Value])
    if snapshotEntry != other.snapshotEntry {return false}
                     ^
/.../Work/Swift/Systems/****/.build/checkouts/***-5523697989699535816/Sources/***Messages.pb.swift:1476:17: error: binary operator '!=' cannot be applied to two '[Proto****]' operands
    if trendbar != other.trendbar {return false}
       ~~~~~~~~ ^  ~~~~~~~~~~~~~~

Any help appreciated!

swift version: 3.1 protoc version: 3.2.0 protoc-gen-swift version: 0.9.29 MacOS: 10.12.4

Note: This is likely a duplicate of https://github.com/apple/swift-protobuf/issues/1014 but I didn't want to cram all of this into a comment in a long chain. Feel free to close as duplicate if you want.

Summary/TLDR

The change to move computed properties backed by _storage to properties stored directly on the message structs has caused massive size increases for messages at the top of a deep hierarchy. The switch code generated for oneof parsing is sensitive to these size increases, leading to massive stack frames. In debug mode without optimization these stack frames can easily exhaust the 512KB stack of a background dispatch thread and cause a crash.

Details

We are experiencing a crash in debug builds after upgrading SwiftProtobuf and protoc-gen-swift from 1.7 to 1.10.2. Our crash happens in the decodeMessage(decoder:) method of a message containing a oneof with lots of cases. From what I can tell the stack frame generated for the withExtendedLifetime closure in that method is huge, about 800KB. Since the decoding is happening on a background thread with a 512KB stack, the app crashes as soon as this closure is invoked.

The message is of the form:

message RenderDataDTO {
  SomeMessage1 m1 = 1;
  SomeMessage2 m2 = 2;
  SomeMessage3 m3 = 3;
  oneof render {
    SomeCaseMessage1 case1 = 100;
    SomeCaseMessage2 case2 = 101;
    SomeCaseMessage3 case3 = 102;
    // 20 cases in total
    SomeCaseMessage20 case20 = 120;
  }

The code that gets generated for decodeMessage(decoder:) has the form:

  public mutating func decodeMessage<D: SwiftProtobuf.Decoder>(decoder: inout D) throws {
    _ = _uniqueStorage()    // LINE A
    try withExtendedLifetime(_storage) { (_storage: _StorageClass) in    // LINE B
      while let fieldNumber = try decoder.nextFieldNumber() {
        switch fieldNumber {
        case 1: try decoder.decodeSingularMessageField(value: &_storage._m1)
        case 2: try decoder.decodeSingularMessageField(value: &_storage._m2)
        case 3: try decoder.decodeSingularStringField(value: &_storage._m3)
        case 100:
          var v: SomeCaseMessage1?
          if let current = _storage._render {
            try decoder.handleConflictingOneOf()
            if case .case1(let m) = current {v = m}
          }
          try decoder.decodeSingularMessageField(value: &v)
          if let v = v {_storage._case1 = .case1(v)}
        case 101:
          var v: SomeCaseMessage2?
          if let current = _storage._render {
            try decoder.handleConflictingOneOf()
            if case .case2(let m) = current {v = m}
          }
          try decoder.decodeSingularMessageField(value: &v)
          if let v = v {_storage._case2 = .case2(v)}
       // etc...

If I set a breakpoint at the line marked LINE A and check the stack pointer, then set a breakpoint at LINE B and do the same I find that the difference is about 800KB.

Here's one example:

RenderDataDTO.decodeMessage(decoder:)
       rsp = 0x00007000087cdb60

In closure:
       rsp = 0x0000700008704780   <-- 800KB jump!

I have a couple of observations to explain this:

Observation 1:

In the earlier version of swift-protobuf almost all fields of Message structs were stored in _storage, a reference-typed container, and exposed as computed vars. In the current version they seem to much more often (but not always?) be implemented as properties on the message itself. Thus the size of a leaf message struct with no child messages became O(n) instead of O(1), where n is the number of fields in the message.

The story gets a lot worse when you have child messages. Since messages are structs, children get embedded directly in their parents, so the size of a non-leaf message grows like the sum of its entire message sub-tree. So the size scales more like O(n^m), where n is the average number of children per level and m is the average number of levels in the message hierarchy. Since Swift always likes to allocate structs on the stack, this can lead to massive stack frames.

This agrees with what I saw when I measured the size of the SomeCaseMessageN messages using MemoryLayout<SomeCaseMessageN>.size. In the older code, they were all 24 bytes, with a few exceptions. The largest was 80 bytes. In the new code the sizes tended to increase by 10X-20X. 224 and 448 bytes were typical sizes, with several messages growing above 1000B. One grew to 5256 bytes, while the biggest was 6080 bytes.

Observation 2:

If the compiler is smart, it will know that all the cases in the switch statement inside that closure are mutually independent. Only one of them will ever execute per invocation, so their local variables can share space on the stack, and the size of the stack frame will be the size of the biggest case in the switch. Since we're in debug mode it might not be making that optimization, and instead allocating separate space for every case in the switch. Given the size impact of the changes I discussed above, this could add up to a lot of space.

In fact, when I change the optimization mode from none to size or speed, the message sizes don't change but the size of the stack frame drops to about 26KB. That's still an oversized stack frame eating 5% of the stack in release mode, but at least it's not bigger than the entire stack!

For reference, with the code generated from 1.7, the stack frame is about 16KB with optimization mode none. With optimization mode speed the stack frame drops to just 400B.

Wrapping up

There seem to be problems with putting so many stored properties on messages. Perhaps this could be dialed back to where it was before?

Environment: OS: MacOS Catalina 10.15.6 Xcode: Version 11.6 Swift 5

iOS 13.4 Xcode Version 11.4 (11E146) Swift 5 Swift-Protobuf 1.8.0 proto2

My iOS app is using generated code from outdated proto files. Since I've compiled the .swift files from the old protos, we've added some new cases to one of our enums, and started returning the cases to the client from the backend. I'm getting an error when parsing the JSON data into the Swift Proto objects: unrecognizedEnumValue.

From my understanding of the proto spec, it should be fine to add new enum cases, and clients who don't have the latest spec map the unknown value to the 0 case. Is there something wrong with how we name our 0th case?

Old Proto

message Product {
    enum State {
        State_INVALID = 0;
        ACTIVE = 1;
        DISCONTINUED = 2;
    }
    optional State state = 1;
}

New Proto

message Product {
    enum State {
        State_INVALID = 0;
        ACTIVE = 1;
        DISCONTINUED = 2;
        FOR_SALE = 3;
        NOT_FOR_SALE = 4;
    }
    optional State state = 1;
}

JSON Being Parsed

{
    "state": "FOR_SALE"
}

Swift JSON Decoding (using Old Proto)

do {
    var options = JSONDecodingOptions()
    options.ignoreUnknownFields = true
    return try Product(jsonUTF8Data: data, options: options)
} catch {
    print(error) // error is printed unrecognizedEnumValue
    return nil
}

Resolves #1245

This boxes all references to CustomDebugStringConvertible and debugDescription in #if checks to exclude them from the release binary. The existence of this conformance and default implementation on the Message protocol (and some other types) adds up quite considerably. Across our large iOS app, we can attribute 6.7 MB of the uncompressed binary size to generated swift protos. This change cuts that back by 780 KB (11.6%). Other efforts like #1240 will probably also help trim this back even further.

I tried to do this with as little diff as possible. Added some inline comments on some things I'm unsure about.

Special thanks to @bubski for the idea.

tl;dr: Is it possible for this repo to offer a canonical SwiftProtobuf plugin for SwiftPM, as demonstrated here?

Now that Swift 5.6 is out in the Xcode 13.3 betas and later, Swift Packages now support build tool plugins, as defined in SE-0303 and SE-0325.

However, for the time being, it seems like everyone will have to roll their own protobuf to Swift generator plugin. It doesn't seem like there is a plugin already "out there" and one does not simply copy and paste the example from the proposal. Speaking for myself, I am not that familiar with the tool and I would certainly have no idea how to support a hand rolled plugin going forward.

Is it possible for this repo to offer a canoncial SwiftProtobuf plugin for SwiftPM? All I really want to do is declare a dependency to this repo and add a plug in to my target. Is this straightforward usability naive?

Some details about my use case. I just have some protobuf files organised into folders that I want to generate Swift from. I want to compile that source into a library. This library is a dependency among other targets in different packages.

macOS Monterey 12.2 (21D49) Xcode 13.3 / Build version 13E5086k swift-driver version: 1.44.2 Apple Swift version 5.6 (swiftlang-5.6.0.320.8 clang-1316.0.18.8) Target: x86_64-apple-macosx12.0 https://github.com/apple/swift-protobuf/releases/tag/1.18.0

Use the official swift docker images to execute make build test on pull requests targeting master. Creates a test matrix for swift and protobuf versions, currently running tests on both swift 5.1.5 and 5.2 with protobuf 3.11.4. The sub-targets of the test target are broken into steps to allow easier diagnostics and caching of the conformance test runner.

I tried to get caching of the conformance-test-runner working, but I haven't seen it work yet and I'm not sure why.

To see this working: https://github.com/toffaletti/swift-protobuf/pull/1/checks

Unfortunately I wasn't able to use the #if compiler(>=5.0) flag since the project supports older swift versions where this directive is not available.

Motivation

SPM is providing new capabilities for tools to expose themselves as plugins which allows them to generate build commands. This allows us to create a plugin for SwiftProtobuf which invokes the protoc compiler and generates the Swift code. Creating such a plugin is greatly wanted since it improves the usage of the protoc-gen-swift plugin dramatically. Fixes https://github.com/apple/swift-protobuf/issues/1207

Modification

This PR adds a new SPM plugin which generates build commands for generating Swift files from proto files. Since users of the plugin might have complex setups, I introduced a new swift-protobuf-config.json file that adopters have to put into the root of their target which wants to use the new plugin. The format of this configuration file is quite simple:

{
    "invocations": [
        {
            "protoFiles": [
                "Foo.proto"
            ],
            "visibility": "internal"
        }
    ]
}

It allows you to configure multiple invocations to the protoc compiler. For each invocation you have to pass the relative path from the target source to the proto file. Additionally, you have to decide which visibility the generated symbols should have. In my opinion, this configuration files gives us the most flexibility and more complex setups to be covered as well.

Open topics

Hosting of the protoc binary

Hosting of the protoc binary is the last open thing to figure out before we can release a plugin for SwiftProtobuf. From my point of view, there are three possible routes we can take:

1. Include the artifactbundle inside the SwiftProtobuf repository
2. Include the artifactebundle as an artifact on the GH releases in the protobuf repo
3. Extend the the artifact bundle manifest to allow inclusion of URLs. This would require a Swift evolution pitch most likely.

However, with all three of the above we would still need to release a new version of SwiftProtobuf with every new release of protoc.

Future work

Proto dependencies between modules

With the current shape of the PR one can already use dependencies between proto files inside a single target. However, it might be desirable to be able to build dependency chains of Swift targets where each target includes proto files which depend on protoc files from the dependencies of the Swift target. I punted this from the initial plugin because this requires a bit more work and thinking. Firstly, how would you even spell such an import? Secondly, the current way of doing ProtoPathModuleMapping files is not super ideal for this approach. It might make sense to introduce a proto option to set the Swift module name inside the proto files already.

Result

We now have a SPM plugin that can generate Swift code from proto files. To use it, it provides a configuration file format to declare different invocations to the protoc compiler.

TODOs:

• As soon as we agreed on the binary problem, I am going to write documentation how to use the plugin

Hi! Could you add support for swift 5.1?

By the way, how could I disable all checks for swift 4.2 in generated proto files? I would like to have only latest swift version in generated proto files.

Thanks!

OS: macOS 10.13.3 Xcode version: 9.3 beta 4 Swift version: 4.1 SwiftProtobuf version: 1.0.3

I previously posted this issue in the grpc-swift project (here) but MrMage referred me to the swift-protobuf project since I mistakenly assumed it was a problem with grpc-swift.

In a *.proto file we are defining a custom method option in the following way:

extend google.protobuf.MethodOptions {
    bool requiresAuthentication = 50001;
}

When generating the Swift code an import for SwiftProtobufPluginLibrary is missing in the generated *.pb.swift file. Google_Protobuf_MethodOptions is being extended in the generated code but it is being marked as an undeclared type until I manually import SwiftProtobufPluginLibrary in addition to the already imported SwiftProtobuf. This is fixing the issue but since ideally the generated code is never modified manually this is not a great solution.

Is this a known issue and/or has somebody encountered this issue before?

• Implicit Sendable conformance for all Message and Enum.
• Generated Sendable conformance for oneof enums.
• Generated @unchecked Sendable for messages with heap storage.
• Explicit Sendable conformance for extension-related tooling.

Notes:

• Tested by applying the changes to a medium-sized codebase that has pervasive oneof and heap storage, but few uses of advanced features like extensions or RPC.
• The code is formulated to be pre-5.5 compatible, while avoiding putting two copies of the entire type definition inside #if. However, I don't have an older compiler handy to check.
• I left types and protocols used purely during serialize/deserialize alone, like ExtensionMap, _NameMap, [Blah][De|En|codingOptions, etc.
• I didn’t bump the compatibility version since it doesn’t technically create API or ABI incompatibilities — just warnings, for now. I don’t know if this was the right call.

Tasks:

• [x] Investigate generating extension Blah: @unchecked Sendable {} unconditionally for messages, enums, and oneof types to avoid having to bump the API compatibility version.
• [x] Update generated comments: https://github.com/apple/swift-protobuf/pull/1208#discussion_r801482071

iOS: 16.x Swift: 5.7 protoc: 3.21.9 SwiftProtobuf: 1.17.0

Should this unit test pass?

func testHashValuesAreNotEqualBetweenDifferentUnderlyingValues() {

    let myComponent1 = MyComponent.with { component in
        component.titles = ["Hello", "World"]
    }

    let myComponent2 = MyComponent.with { component in
        component.titles = ["Ipsum", " Lorem"]
    }

    let any1 = try! Google_Protobuf_Any(message: myComponent1)
    let any2 = try! Google_Protobuf_Any(message: myComponent2)

    XCTAssertNotEqual(any1.hashValue, any2.hashValue)
}

Hello,

I would like to keep the generated files in the repository as well as the .proto files. This seems to be useful at least for the following reasons:

• The protoc is not part of the project, and its version is something that is defined by the environment. I want to be sure that some strange CI environment (with old or buggy version of the protoc) will not ruin my production.
• Check and monitor any change to the generated files (who did when and why), especially in case of protoc version change.

The current version doesn't work for me as it puts the generated files into the .build folder.

I would like to either:

• be able to specify the path for the generated file, and make it possible to put it to the source path.
• have a command to generate swift files from protobuf files.

It seems that https://github.com/apple/swift-evolution/blob/main/proposals/0332-swiftpm-command-plugins.md allows both:

• PluginPermission allows to write to the package directory
• CommandPlugin allows to have a plugin command to perform any action.

The second approach is preferable for me as it allows to run the generation on-request from the developer.

I'm using xCode 14.1 on macbook pro silicon M1 with Ventura 13.0 (22A380).

I'm using Protobuf 1.20.2 via SPM

I receive the following error in the xcode console while indexing.

`Showing All Errors Only

Indexing

/Applications/Xcode-14.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc -swift-version 5 -I /Applications/Xcode-14.1.0.app/Contents/PlugIns/IDESwiftPackageCore.framework/Versions/A/Frameworks/SwiftPM.framework/SharedSupport/ManifestAPI -sdk /Applications/Xcode-14.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk -package-description-version 5.6.0 -parse-as-library -I /Applications/Xcode-14.1.0.app/Contents/PlugIns/IDESwiftPackageCore.framework/Versions/A/Frameworks/SwiftPM.framework/SharedSupport/PluginAPI /Users/me/Library/Developer/Xcode/DerivedData/MyProject-etyfxtttxpmavldwvibjxmvtbbhn/SourcePackages/checkouts/swift-protobuf/Plugins/SwiftProtobufPlugin/plugin.swift -index-file -index-file-path /Users/me/Library/Developer/Xcode/DerivedData/MyProject-etyfxtttxpmavldwvibjxmvtbbhn/SourcePackages/checkouts/swift-protobuf/Plugins/SwiftProtobufPlugin/plugin.swift -disable-batch-mode -o /Users/me/Library/Developer/Xcode/DerivedData/MyProject-etyfxtttxpmavldwvibjxmvtbbhn/SourcePackages/checkouts/swift-protobuf/Plugins/SwiftProtobufPlugin/plugin.swift.o -index-store-path /Users/me/Library/Developer/Xcode/DerivedData/MyProject-etyfxtttxpmavldwvibjxmvtbbhn/Index.noindex/DataStore

error: module name "plugin.swift" is not a valid identifier; use -module-name flag to specify an alternate name
error: Command failed with exit code 1`

This leads to xCode not linting code, jump to definition not working and other minor indexing issues.

Any help?

Thank you

Note: only the last three commits are actually related to this change. Previous commits are actually part of https://github.com/apple/swift-protobuf/pull/1326, on which this PR depends.

Motivation

This is a follow-up PR for the removal of Data (and eventually Foundation) from swift-protobuf. This is related to https://github.com/apple/swift-protobuf/issues/816 and follows the changes made in https://github.com/apple/swift-protobuf/pull/1326

Modifications

This PR includes the bulk of the required changes for the removal of Data from swift-protobuf:

• Changes to the codegen, to move away from using Data and use [UInt8] instead for byte-type fields.
• Regenerated protos based on the codegen changes.
• Changes to remove Data completely from the binary deserialisation path.
• Other required changes to make things compile with the new generated code.

Other PRs will follow, removing Data usages from the binary serialisation path, the JSON (de)serialisation, and any other miscellaneous remaining usages.

Result

The codegen and binary deserialisation paths are now Data-free.

As mentioned in another issue, we have our proto files integrated using submodules. We want to generate code for proto files in specific directories in this submodule. in the protoc command, we can do this with glob patterns, and it'd be great to be able to support this in the plugin so that we don't need to have a large list of files, if possible.

Our use case requires a pattern that looks like this:

submodules/**/feature/*.proto

Alternatively, if glob's aren't possible, adding a directory that holds proto files would allow the config to be more lean and easier to maintain.

Motivation

The Swift compiler added @_spi() public visibility annotations in Swift 5.3 which allows you to create a system programming interface. This is often preferred to be used over @testable since @testable only works in debug modes or when passing specific compiler flags. Right now protoc-gen-swift only allows to generate internal or public visibility levels. However sometimes you want to use the generated proto types also in your test without public and without @testable

Modification

Added a new visibility modifier to the code gen plugin called _SPI(foo). I intentionally made it underscored and did not document it since it is not a stable annotation. Users that reach out to use it should know what they are doing.

Result

You can now generate types with @_spi(foo) public access level.