After some experiments I found a bug. It seems to be because I use swift-numerics in the function. https://colab.research.google.com/drive/1TL70U_I_2wEIyzSa5quB55h54A79bDtp?usp=sharing

There's an easy way to restart the runtime without the arduous task of clicking the menu! Found on StackOverflow.

import os
os.kill(os.getpid(), 9)

I also added some comments to explain the benign error messages that appear.

I'm not sure where to PR the Colab notebook itself.

Swift 5.6.1, Swift-Colab Install Script Latest Release (v2.0), Google-Colab.

I tried to test concurrency, but it fails to launch anything. Creating any Task (structured or detached) on top level crashes the runtime.

First I imported Foundation and _Concurrency. Without _Concurrency it just doesn't recognise Task.

import Foundation
import _Concurrency

And then launching one of the following crashes:

Task {
    for i in 1...100{
        print(i)
        try? await Task.sleep(nanoseconds: 1_000_000_000)
    }
}

Task {}

Task.detached {}

I don't know if it will help, but here's the log I got. It seems like it just contains bunch of restart messages.

app.log

I have seen this bug around for quite some time now. IIRC, the bug existed when Google sponsored S4TF. Pandas DataFrames are having their columns rearranged before rendering to the Colab GUI. This happens roughly 50% of the time that you render a DataFrame:

Code to reproduce:

%install-swiftpm-flags $clear
%install '.package(url: "https://github.com/pvieito/PythonKit.git", .branch("master"))' PythonKit
%install '.package(url: "https://github.com/KarthikRIyer/swiftplot", .branch("master"))' SwiftPlot AGGRenderer
%include "EnableIPythonDisplay.swift"
import Foundation
import PythonKit
import SwiftPlot
import AGGRenderer

let display = Python.import("IPython.display")
let pd = Python.import("pandas")
display.display(pd.DataFrame.from_records([["col 1": 3, "col 2": 5], ["col 1": 8, "col 2": 2]]))

This could be a symptom of feeding the data through LLDB. If it was serialized through JSON with my new inter-process piping mechanism, perhaps it would always render correctly.

When building Swift packages, there are multiple problems:

1. The formatting from build output could leak out and make a hard-coded message about Clang modules turn green.
2. There is an error of build.db missing when compiling a Swift package with the --verbose flag, at least with S4TF.
3. Two module.modulemap files that declare the same Clang module can overwrite each other, even if one is part of the documentation of a Swift package and never actually involved in the build process. This happened with the modulemap currently in the Utilities directory of s4tf/s4tf.
4. The headers referenced by a modulemap file are absent when you re-load the package. The -Xcc -I/... flags from when you built the package helped SwiftPM locate headers. Now that LLDB is loading the module with no knowledge of those manual includes, it doesn't know where the headers are.
5. https://github.com/apple/swift/issues/58916 is still an issue and I don't have a workaround for it.

For some of these problems, a workaround may not be possible or feasible. Thus, I might have to make documentation warning the user to copy certain header files to the system include path. Or, I might have to warn against adding duplicate modulemap files to a GitHub repo and anything it recursively depends on.

I was working on reproducing a crash in swift-reflection-mirror, and successfully reproduced it on a non-Apple platform. The stack trace for the runtime error has slightly incorrect formatting. I never accounted for this type of error message while making Swift-Colab, which is probably why it's improperly formatted. However, I can at least use the stack trace to investigate the crash in ReflectionMirror.

// Cell 1 - pulls from the "colab-crash-1" branch
%install '.package(url: "https://github.com/philipturner/swift-reflection-mirror", .branch("colab-crash-1"))' ReflectionMirror
@_spi(Reflection) import ReflectionMirror
print(_forEachField)

// Cell 2
struct ASimpleKPI {
  var w = 1
}

struct AMixedKPI {
  var string = "foo"
}

struct ANestedKPI {
  var simple = ASimpleKPI()
  var mixed = AMixedKPI()
}

// Cell 3
var x = ANestedKPI()
    
do {
  var result: [PartialKeyPath<ANestedKPI>] = []
  
  var out = [PartialKeyPath<ANestedKPI>]()
  _forEachFieldWithKeyPath(of: ANestedKPI.self, options: .ignoreUnknown) { _, kp in
    out.append(kp)
    return true
  }
  
  for kp in out {
    result.append(kp)
    if x[keyPath: kp] is ASimpleKPI {
      _forEachFieldWithKeyPath(of: ASimpleKPI.self, options: .ignoreUnknown) { _, nkp in
        result.append(kp.appending(path: nkp as AnyKeyPath)!)
        return true
      }
    } else if x[keyPath: kp] is AMixedKPI {
      var out2 = [AnyKeyPath]()
      _forEachFieldWithKeyPath(of: AMixedKPI.self, options: .ignoreUnknown) { _, nkp in
        out2.append(nkp as AnyKeyPath)
        return true
      }
      
      for nkp in out2 {
        result.append(kp.appending(path: nkp)!)
      }
    }
  }
}

// Cell 4
do {
  var result: [PartialKeyPath<ANestedKPI>] = []
  
  var out = [PartialKeyPath<ANestedKPI>]()
  _forEachFieldWithKeyPath(of: ANestedKPI.self, options: .ignoreUnknown) { _, kp in
    out.append(kp)
    return true
  }
  
  for kp in out {
    result.append(kp)
    if x[keyPath: kp] is ASimpleKPI {
      _forEachFieldWithKeyPath(of: ASimpleKPI.self, options: .ignoreUnknown) { _, nkp in
        result.append(kp.appending(path: nkp as AnyKeyPath)!)
        return true
      }
    } else if x[keyPath: kp] is AMixedKPI {
      _forEachFieldWithKeyPath(of: AMixedKPI.self, options: .ignoreUnknown) { _, nkp in
        result.append(kp.appending(path: nkp as AnyKeyPath)!)
        return true
      }
    }
  }
}

Run the cells in this order:

• Cell 1
• Cell 2
• Cell 3
• Cell 4
• Cell 3
• Cell 4

In the second run of Cell 4, there should be a runtime crash. The error message's second stack frame says ReflectionMirror.swift/, with an extraneous slash after swift. The bug occurs in the function prettyPrintStackTrace, inside the file FormatErrors.swift. Permalink here.

On second examination, the first frame is also incorrect at closure #3 in. There are two spaces between in and -, when I would like only one present.

Swift-Colab uses the Python pexpect library to interact with running processes. This lets it extract colorized, interactive output - a feature not present in google/swift-jupyter. However, this means it will freeze when the process requests input.

In the image above, the third %system command requests permission to overwrite a file.

There is currently no mechanism to pass input into a running process. The Python Jupyter kernel passes input in, so it is theoretically possible. But this feature will take a non-negligible amount of time to implement. I am deferring its addition to a future release.

Although %system commands now support user input, this wasn't extended to the LLDB process. Swift code can write to output with print(), but can't read through readLine(). Any libraries that call this function will be affected.

As an alternative to this kind of interaction, ask the user to paste a string into an empty cell. Design the notebook with a drop-down feature that closes the cell, or ask the user to delete it after running. For larger input, the user can connect to their Google Drive to import .txt or .csv files.

Seems we are the only two people use Swift with notebooks. Anyway, I think we need to collaborate on a new header-like library called JupyterDisplay. I have a prototype: https://github.com/liuliu/swift-jupyter/blob/main/display/JupyterDisplay.swift

It is really simple. The idea is that any other libraries can depend on this library and provide notebook rich display support. If you looked at Python packages, many of them "battery-included" for notebook displays either inside the main package or in the support package.

So far, the EnableJupyterDisplay.swift implementation took hands into itself and provides backends for matplotlib or Python or SwiftPlot. That is really limited.

On the technical side, this package simply exposes a JupyterDisplay instance that EnableJupyterDisplay.swift can switch to a different implementation, thus, enabling the said behavior.

Let me know what do you think. I mainly want to propose moving this into a "neutral" org and package it lightly (like package to SwiftPM and Bazel) and everyone can use.

In the recent Swift Numeric call, the idea of lowering the bar for scripting in Swift in the context of Data Science was surfaced, which lead to a suggestion to see if we could dockerize swift-colab.

From reading through the swift-colab repo, I see the following possible directions:

• package the install process as a set of binaries which can be directly downloaded in colab instead of compiling it from scratch
• the above should also be usable in a standard Jupyter(Lab) environment
• dockerize versions of the above as an image that can be used in a JupyterHub environment

Each of the above would lower the bar of using swift-colab, as well as allow use in stand alone Jupyter environments.

Were you thinking about something else?

The main reason I made the overhauls present in Swift-Colab 2.0 was so that in the future, I could run S4TF code without facing bottlenecks that make it virtually unusable. However, I am unable to compile S4TF for use in the interactive experience. This is after avoiding the problems described in #14.

The test notebook S4TF with TF 2.4 shows my effort to compile S4TF for use in the Swift interpreter. Even though that failed, I can technically compile it using %system flags like in s4tf-on-colab-example-1.ipynb and add custom code to the test suite. But that isn't ergonomic or reproducible in any way.

Specifically, the debugger shows an error when I run the following code. Back in the swift-jupyter era, the TensorFlow module was embedded in the toolchain. So the error below was likely never encountered.

import TensorFlow
print(Tensor<Float>.self)

<Cell 1>:2:7: error: cannot find 'Tensor' in scope
print(Tensor<Float>.self)
      ^~~~~~