This is a ⚠️ SUPER EARLY ⚠️ proof of concept, but I have the very basics of SourceKitten building with SwiftPM and linking against libsourcekitdInProc.so on Linux, building with the head of Swift.

Now this is a big mess at the moment because of the simultaneous moves to Swift 3, Linux and SwiftPM changes among other things, but I wanted to start the conversation now.

This branch will likely be frequently force-pushed, be warned.

Here are the (long and tedious) steps to get this running on a fresh x64 Ubuntu 16.04 machine:

$ apt-get update
$ apt-get install git cmake ninja-build clang python uuid-dev libicu-dev icu-devtools libbsd-dev libedit-dev libxml2-dev libsqlite3-dev swig libpython-dev libncurses5-dev pkg-config libblocksruntime-dev autoconf libkqueue-dev libtool
$ git clone https://github.com/apple/swift.git
$ cd swift
$ utils/update-checkout --clone
$ cd ../swift-corelibs-libdispatch
$ git checkout 245c9b3
$ git submodule update --init
$ cd -
$ utils/build-toolchain local.swift
$ utils/build-script --libdispatch
$ git remote add jp https://github.com/jpsim/swift.git
$ git fetch jp
$ git checkout sourcekit-linux
$ rm ../build/Ninja-DebugAssert/swift-linux-x86_64/CMakeCache.txt
$ utils/build-script --libdispatch
$ cd ..
$ git clone https://github.com/jpsim/SourceKitten.git
$ cd SourceKitten
$ git checkout jp-wip-linux
$ export PATH="/root/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64/usr/bin:$PATH"
$ export LINUX_SOURCEKIT_LIB_PATH="/root/build/Ninja-DebugAssert/swift-linux-x86_64/lib"
$ swift build
$ .build/debug/sourcekitten syntax --text "import Foundation"

Which will then print the JSON-formatted syntax map for main.swift. That's the only thing that's exposed via the CLI interface at the moment. Tests don't run either.

At this point, I think the next steps are roughly:

1. ~~Re-add as much of the removed or commented out code that doesn't depend on external packages.~~ (July 18 edit: done)
2. ~~Add Swift 3 & Linux support for upstream dependencies (Commandant (jp-05-31-snapshot), Result, YamlSwift (jp-wip-linux), SWXMLHash (jp-swift-3) & Clang_C). I've started this process for some of these, but they're all very hacked together and/or out of date at the moment.~~ (June 30 edit: done)
3. ~~Re-integrate dependencies.~~ (June 30 edit: done)
4. Minimize differences between SourceKitten's master branch and the Linux one.
5. Legitimately enable SourceKit on Linux upstream in the apple/swift repo.
6. ~~Add CI integration for Linux.~~ (July 18 edit: done)

We may also want to stage this out, for example by first getting Swift 3 support on OS X merged in, then maybe moving to a SwiftPM-based workflow, dropping Carthage & generating the Xcode project via swift package generate-xcodeproj.

Given the massive changes in Swift 3, I'd probably want to drop Swift 2.x support even before Xcode 8 GM is out.

@norio-nomura are you interested in helping with any of this?

Hello friends! Trying to update Slang to run with the latest build and stumbling across a very bizarre crash inside the test suite when getting file lines:

This doesn't happen with 0.26.0 and 0.27.0, only in 0.28.0 and above, where StringView was introduced, I'm just guessing that this has something to do with it. I know that this is most likely a glitch within DispatchQueue inside XCTest bundle, I've seen some weirdness in the past. For example, this almost doesn't happen (very-very rarely) when running a single test, but occurs solidly when running all tests. At the same time, 0.27 doesn't look that much different to 0.28, however, the crash doesn't happen there. It doesn't occur at the same place, but always when calling _contentsQueue.sync inside _stringViewQueue.sync. This isn't a deadlock, but looks very much like some race condition…

I'm wondering if you've dealt with anything like that before? Any advice?

Also, is there any reason why SourceKitten uses queues for atomic ops and not locks? Is that a Linux-related issue or is there a different reason?

Hi,

When running XCode with SOURCEKIT_LOGGING=3 I can see some diagnostic information being returned in the response:

  key.diagnostics: [
    {
      key.line: 3,
      key.column: 16,
      key.description: "'loadItemForTypeIdentifier(_:options:completionHandler:)' has been renamed to 'loadItem(forTypeIdentifier:options:completionHandler:)'",
      key.filepath: "/Users/alvaro/test/test.playground",
      key.severity: source.diagnostic.severity.error,
      key.fixits: [
        {
          key.offset: 34,
          key.length: 55,
          key.sourcetext: "loadItem"
        }
      ],
      key.diagnostic_stage: source.diagnostic.stage.swift.sema
    }
  ]

Im interested in getting this JSON but it doesnt seem to be included in any of the built-in commands. Could it be possible to include a new command to get this information?

Thanks, A

Analogous to #150, but for hacking on SourceKitten under Xcode.

Builds and runs in Xcode 6.2 and 6.3, but currently the tests only succeed under 6.2 with any Swift prior to 2016-01-25. Later builds of Swift 2.2 have the new associatedtype token and Xcode 6.3 causes changes to the autocomplete output that are valid (better, even) but don't pass the tests.

Another approach for fixing #167

Improved from #172:

• No execv
• SPM compatible
• Searching toolchain path is extended by applying reviews on #172.

Progress:

• [x] sourcekitd.framework
• [x] libclang.dylib

Add SourceKitVariant that replacing SourceKitRepresentable.

• Replace complicated SourceKitRepresentable's operations to simple operations
• No performance penalty
  • Same on whole data parsing
  • Better on partial data parsing

func walk1(dictionay: [String: SourceKitRepresentable]) {
    if let _ = dictionay[SwiftDocKey.name.rawValue] as? String {
    }
    if let _ = dictionay[SwiftDocKey.kind.rawValue] as? String {
    }
    if let _ = (dictionay[SwiftDocKey.offset.rawValue] as? Int64).map({Int($0)}) {
    }
    if let _ = (dictionay[SwiftDocKey.length.rawValue] as? Int64).map({Int($0)}) {
    }
    guard let substructures = dictionay[SwiftDocKey.substructure.rawValue] as? [SourceKitRepresentable] else { return }
    for substructure in substructures {
        if let substructure = substructure as? [String: SourceKitRepresentable] {
            walk1(dictionay: substructure)
        }
    }
}

will be changed to:

func walk2(variant: SourceKitVariant) {
    if let _ = variant[.name]?.string {
    }
    if let _ = variant[.kind]?.string {
    }
    if let _ = variant[.offset]?.int {
    }
    if let _ = variant[.length]?.int {
    }
    guard let substructures = variant[.substructure]?.array else { return }
    for substructure in substructures {
        walk2(variant: substructure)
    }
}

Kitten reports the offset for the function body to be off by 1, and chops off the beginning of the function name for the body:

unc numberOfComponents(in pickerView: UIPickerView) -> Int { return 1 }

Was parsing BitcoinTicker on Github and noticed all the function bodies are missing the first character.

A colleague and I tried to install SourceKitten as a Swift Package Manager dependency earlier this week...

(Historical reenactment.)

Unfortunately, the current workarounds in SourceKitten's Makefile don't propagate to dependencies. While we were able to see some success by copying over some of the -Xcc and -Xlinker flags, heavily editing and paring down a forked version of the SourceKitten repo, and specifying sourcekitd and clang-c dependencies explicitly in the Package.swift, in the end we couldn't get it to work.

We ended up running out of time, so we built the tool into an OS X app instead (where SourceKitten—and other dependencies—installed flawlessly). I did want to flag the lack of SPM support as a current issue, though.

There's one other thing I'd love to have discussion around after diving into the SourceKitten code base: have you thought about dividing SourceKitten's responsibilities into multiple libraries? The sourcekitd bits don't seem to require the clang-c bits at all—the latter seems to be used for documentation support/analyis and the tool we were working on really only needed the former. We may have been able to get things up and running on SPM quickly if it weren't for the Clang_C module.

Tried to build a CLI and including kitten in a pod ... built ok, won't run due to what seems like the inability to statically link frameworks (or reference frameworks) from a CLI.

Can this be built to run / be called from a CLI app and run on Ubuntu ?

When I use jazzy to parse my project, but encounter such error:

fatal error: unexpectedly found nil while unwrapping an Optional value
    from /Library/Ruby/Gems/2.0.0/gems/jazzy-0.2.1/lib/jazzy/sourcekitten.rb:84:in `run_sourcekitten'
    from /Library/Ruby/Gems/2.0.0/gems/jazzy-0.2.1/lib/jazzy/doc_builder.rb:53:in `block in build'
    from /Library/Ruby/Gems/2.0.0/gems/jazzy-0.2.1/lib/jazzy/doc_builder.rb:51:in `chdir'
    from /Library/Ruby/Gems/2.0.0/gems/jazzy-0.2.1/lib/jazzy/doc_builder.rb:51:in `build'
    from /Library/Ruby/Gems/2.0.0/gems/jazzy-0.2.1/bin/jazzy:15:in `<top (required)>'
    from /usr/bin/jazzy:23:in `load'
    from /usr/bin/jazzy:23:in `<main>'

It seems that something goes wrong when in sourcekitten stage. I don't know how to collect more logs and how to diagnose the error. Any suggestion? Thanks!

Since Xcode’s New Build System uses llbuild, compiler arguments for Module can be retrieved from manifest of llbuild instead of parsing xcodebuild’s output. Until this changes, the clean action was used for xcodebuild to ensure compiler arguments, but it is not necessary when using the New Build System.

I tried the complete version from documentation but with my current SDK. sourcekitten complete --text "import UIKit ; UIColor." --offset 22 -- -target arm64-apple-ios16.0-simulator -sdk /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator16.0.sdk

:0: warning: using sysroot for 'MacOSX' but targeting 'iPhone' :0: error: unable to load standard library for target 'arm64-apple-ios16.0-simulator' Error: Loading the standard library failed

I don't know whats wrong. Any ideas?

I recently tried running SwiftLint with --fix --format and saw that it removed thousands of whitespace only lines. It'd be great if the options could mirror the options available in Xcode. Particularly an option to ignore whitespace only lines.

We noticed that there are issues with the listing of Swift name and declaration in the SourceKitten output for Objective-C API that includes forward declarations. When using a forward declaration of e.g. a protocol, the Swift name/declaration is missing completely.

Consider the following two files in a project, and an umbrella header importing both of those:

#import <Foundation/Foundation.h>

@protocol PSPDFProtocol
@end

#import <Foundation/Foundation.h>

NS_ASSUME_NONNULL_BEGIN

@protocol PSPDFProtocol;

@interface PSPDFClass

- (id<PSPDFProtocol>)protocolAPI;

@end

NS_ASSUME_NONNULL_END

The SourceKitten doc command results in the following output:

[
  {
    "PSPDFClass.h" : {
      "key.diagnostic_stage" : "",
      "key.substructure" : [
        {
          "key.always_deprecated" : false,
          "key.always_unavailable" : false,
          "key.deprecation_message" : "",
          "key.doc.column" : 12,
          "key.doc.file" : "PSPDFClass.h",
          "key.doc.line" : 7,
          "key.filepath" : "PSPDFClass.h",
          "key.kind" : "sourcekitten.source.lang.objc.decl.class",
          "key.name" : "PSPDFClass",
          "key.parsed_declaration" : "@interface PSPDFClass\n\n- (id<PSPDFProtocol>)protocolAPI;\n\n@end",
          "key.parsed_scope.end" : 11,
          "key.parsed_scope.start" : 7,
          "key.substructure" : [
            {
              "key.always_deprecated" : false,
              "key.always_unavailable" : false,
              "key.deprecation_message" : "",
              "key.doc.column" : 22,
              "key.doc.file" : "PSPDFClass.h",
              "key.doc.line" : 9,
              "key.filepath" : "PSPDFClass.h",
              "key.kind" : "sourcekitten.source.lang.objc.decl.method.instance",
              "key.name" : "-protocolAPI",
              "key.parsed_declaration" : "- (id<PSPDFProtocol>)protocolAPI;",
              "key.parsed_scope.end" : 9,
              "key.parsed_scope.start" : 9,
              "key.unavailable_message" : "",
              "key.usr" : "c:objc(cs)PSPDFClass(im)protocolAPI"
            }
          ],
          "key.swift_declaration" : "class PSPDFClass",
          "key.swift_name" : "PSPDFClass",
          "key.unavailable_message" : "",
          "key.usr" : "c:objc(cs)PSPDFClass"
        }
      ]
    }
  },
  {
    "PSPDFProtocol.h" : {
      "key.diagnostic_stage" : "",
      "key.substructure" : [
        {
          "key.always_deprecated" : false,
          "key.always_unavailable" : false,
          "key.deprecation_message" : "",
          "key.doc.column" : 11,
          "key.doc.file" : "PSPDFProtocol.h",
          "key.doc.line" : 3,
          "key.filepath" : "PSPDFProtocol.h",
          "key.kind" : "sourcekitten.source.lang.objc.decl.protocol",
          "key.name" : "PSPDFProtocol",
          "key.parsed_declaration" : "@protocol PSPDFProtocol\n@end",
          "key.parsed_scope.end" : 4,
          "key.parsed_scope.start" : 3,
          "key.swift_declaration" : "protocol PSPDFProtocol",
          "key.swift_name" : "PSPDFProtocol",
          "key.unavailable_message" : "",
          "key.usr" : "c:objc(pl)PSPDFProtocol"
        }
      ]
    }
  }
]

Note that the swift_name and swift_declaration are missing from the protocolAPI symbol.

One solution to this would be to replace the forward declaration with an actual import of the file that includes the protocol, but in some cases this is not a viable solution as it results in circular dependencies.

Is there any way that the Swift API for this could be generated correctly?

Sorry. This one overlaps (and includes your changes from) my previous pull request but I figured you wouldn't want me updating the previous one after you already approved it.

Refactored Modules.doc computed property to make use of the dispatch queue to speed up the computation. Depending on the number of CPUs/Cores the speed up can be quite dramatic.

I'm trying to generate the documentation for our unit test module, and I am getting an error. It worked for the main module though, and it also worked for the tests a couple of years ago, so not sure what's changed.

My command is: sourcekitten doc --module-name [ProjectName]Test -- -workspace [ProjectName].xcworkspace -scheme "[ProjectName] Dev" clean build > swiftDoc.json

And I get the following error:

Could not parse compiler arguments from `xcodebuild` output.
Please confirm that `xcodebuild` is building a Swift module.
Saved `xcodebuild` log file: /var/folders/8w/w7k_803179l5pbqzm47mxx4c0000gn/T/xcodebuild-2EEBEE83-7072-42FB-A8C6-5688FF202D93.log
Error: Failed to generate documentation

But the xcodebuild log shows "** BUILD SUCCEEDED **" as the last line.

Sourcekitten version: 0.31.1 Xcode version 13.0