We introduced a new AST renderer in https://github.com/iwasrobbed/Down/pull/132

Next, we should create a default Styler which people can easily override so they can get up and running with attributed string rendering

• Follow-up to #132
• Closes #138
• Closes #179

What’s in this PR?

It’s finally here! This PR contains a base implementation of the Styler protocol introduce in https://github.com/iwasrobbed/Down/pull/132. With this default implementation, it will quick and easy to render attributed strings via the AST. My hope is that the implementation is flexible and broad enough to cover the majority of styling needs.

Overview

I did my best to ensure that all block elements preserved the styling of inline elements, which themselves also preserve other inline elements.

List formatting was particularly tricky, but I managed to support right aligned item markers up to a specified width as well as indentation for nested lists.

As a bonus, code blocks are rendered in colored background container, just like in Github (more on that below).

Configurability

The Styler supports three main axes of configurable attributes: fonts, text color and paragraph styling. These different types of attributes are grouped together into collections: FontCollection, ColorCollection, ParagraphStyleCollection. This allows the user to easily specify related attributes in a single place. For example, one font collection could be for static fonts, another for dynamic fonts. Or separate color collections for light and dark mode.

These collections are further grouped together into the DownStylerConfiguration struct, so that different configurations can be swapped for different rendering styles. The configuration also contains a group of other options, used to configure the appearance of the custom attributes.

Custom Attributes

Unfortunately, not all markdown elements can be visually represented using an NSAttributedString. These include the stripes for block quotes, the horizontal rule for thematic breaks, and the inset background container for code blocks. In order to render these elements, I defined some custom attributes.

In order to render these attributes, a custom layout manager is required. The DownLayoutManager can be wired up to an existing text view subclass to support custom attributes, or alternatively one could use the DownTextView, which is a preconfigured TextKit stack that will automatically render markdown content (with custom attributes) whenever the text property is set.

Down Text View & Debug Text View

While working on the implementation I used a variation of DownLayoutManager that draws the boundaries of line fragments. Line fragments are the bounding rectangles in which lines are laid out. The DownDebugLayoutManager will indicate the line rect fragments (the rect containing the line) in red. The used line rect fragments (the rect enclosing the text within a line fragment) are drawn in blue.

I’ve exposed DownDebugLayoutManager and DownDebugTextView because they’re useful tools to help specify the properties affecting paragraph styling.

API Breaking Changes

Some of the API declared in https://github.com/iwasrobbed/Down/pull/132 needed to be adjusted to grant the Styler more information to work with. These include adding arguments for the nest depth for lists and quotes, the prefix length for list items, and a new style method for the list item prefix.

Note

Images aren’t yet supported (they appear just like links), but I’d like to add the support next.

Inline code elements don’t have colored backgrounds yet, but I may add support for that too at a later date.

Finally, having mostly worked with iOS, I developed this primarily from an iOS point of view. Nonetheless, the default styler works across the other platforms too.

What's In This PR?

This PR introduces the ability to traverse the abstract syntax tree generated by cmark, which will lead to the construction of an NSAttributedString without the need to render html in a web view.

• Closes https://github.com/iwasrobbed/Down/issues/134
• Closes https://github.com/iwasrobbed/Down/issues/100
• Closes https://github.com/iwasrobbed/Down/issues/121

How Does It Work?

The Node protocol represents a wrapper around the cmark_node type. There is a concrete class for each type of cmark_node, which exposes the relevant properties for that type.

The Visitor protocol describes an object that is able to traverse the syntax tree. This approach is quite flexible, inasmuch that arbitrary data can be generated from the tree using a concrete implementation of this protocol. For example, we could define two different visitors to generate an NSAttributedString: one that renders the styled text, and another that does the same, but also includes the markdown syntax.

AttributedStringVisitor is a concrete visitor implementation that generates an NSMutableAttributedString. As it visits each nodes it adds the necessary line breaks and textual formatting and applies the visual attributes. It does not apply the attributes directly, but delegates this task to a Styler.

The Styler protocol declares a styling method for each type of node. Each method receives a reference to the NSMutableAttributedString as well as potentially additional information which may be needed (header size, urls etc.).

A concrete Styler can choose exactly how the attributes should be applied. For example, one styler could allow headers to contain bold or italic styles, while another styler could prohibit this by overwriting any such attributes that may have been previously applied to the header.

Notes

I haven't provided a concrete styler as this is quite an involved task and I think it would be best to address this in a separate PR, if you are interested in it. From my experience the hardest part to get right is the paragraph styling for nested blocks such as lists and block quotes.

I would like to update the readme and possibly the demo to elucidate this feature (this could also be done in the separate PR).

Please help prevent duplicate issues before submitting a new one:

• [x] I've searched other open/closed issues for duplicates before opening up this new issue.

We had this issue for example project (https://github.com/iwasrobbed/Down/issues/57) but solution was related to only example project, but not all other cases.

Report

What did you do?

1. Install the framework via carthage or manually to a project
2. Move sources to another directory (remove/rename it). (For carthage - Carthage/Checkout, for manual installing - directory with sources)

What did you expect to happen?

The project is compiled and worked without any errors

What happened instead?

"Missing required module 'libcmark'" error is displayed.

Investigation Why does it happen? Seems like when a framework project contains a module with C code and as result a modulemap to this code, then result framework will contain links to the same C code with absolute paths. You can see that inside Down.framework/Modules/Down.swiftmodule/. Just open any swiftmodule file with any text editor and look for old modulemap path. I mean, this C code/headers(maybe just headers, not sure about that) should be placed at the same place for using the framework, otherwise framework won't find this C-module (in this case - cmark)

Workaround Only workaround what I've found is to copy cmark folder(folder with C code) inside your project (where you have to use Down.framework) and configure Import Paths to this directory. Then, when Down.framework have to find cmark module, it will use the project settings (module is configured and load to the project already). So, either do not remove Checkout directory for carthage (because framework will look for cmark module there) and as result do not use any caches for carthage (like rome), or add cmark directory to your project and configure Import Paths.

Bridging-Header solution If we remove using modulemap and add Bridging-Header to the project, then this case will work only when we don't have Checkouts folder. But at the same time, it won't work if sources are there (if you build it through carthage and do not remove Checkouts directory). In that case you will have Failed to import bridging header error when trying to build your project.

Please help prevent duplicate issues before submitting a new one:

• [X] I've searched other open/closed issues for duplicates before opening up this new issue.

Report

It's just me or everyone also have 5 warnings in the new Xcode 8.3? All of them apparently are in the cmark code btw.

The first 4 apparently can be fixed by removing this 4 files from the build phase, the other one is a type problem

Please help prevent duplicate issues before submitting a new one:

• [x] I've searched other open/closed issues for duplicates before opening up this new issue.

Report

What did you do?

Render a markdown string with an unordered list with a custom font. The custom font is used in the same size for the body as well as the listItemPrefix. The text is rendered via a UILabel.

What did you expect to happen?

The bullet points and list items to be vertically aligned.

What happened instead?

List items with a single line appear further indented than those which have multiple lines.

More

I've looked at the code and it does seem as ListItemParagraphStyler is in charge for formatting all kinds of lists. I do not understand exactly how that is supposed to work tho for unordered lists as it doesn't seem to consider bullet points as a prefix. But I think that alone might be already problematic, in particular when a custom, non-monospaced font is used for listItemPrefix.

On a side note: I've tried to reproduce this in a snapshot test, but I've troubles getting them to run as expected in the first place. First Carthage failed to build the snapshot testing framework in a usable format, now all tests are failing. Any instructions around that?

Please help prevent duplicate issues before submitting a new one:

• [x] I've searched other open/closed issues for duplicates before opening up this new issue.

Report

What did you do?

I am trying to parse the following markdown example:

# First Example

- first item
- second item

# Second Example

- first item

- second item

What did you expect to happen?

Most markdown renderers (e.g. Github) insert a line break between list items in Second Example.

What happened instead?

But Down library does parse both first and second example same way. Here is the output of DebugVisitor:

Document
    ↳ Heading - L1
        ↳ Text - First Example
    ↳ List - type: Bullet
        ↳ Item
            ↳ Paragraph
                ↳ Text - first item
        ↳ Item
            ↳ Paragraph
                ↳ Text - second item
    ↳ Heading - L1
        ↳ Text - Second Example
    ↳ List - type: Bullet
        ↳ Item
            ↳ Paragraph
                ↳ Text - first item
        ↳ Item
            ↳ Paragraph
                ↳ Text - second item

I do understand that you use cmark library for parsing markdown nodes, but maybe I am missing something? Maybe some configuration options?

Or should I report this issue to cmark library repo?

These warnings are thrown by the compiler when including Down as a cocoapod:

By excluding files that should not be compiled from spec.source_files, we can prevent those warnings.

After investigation, I've tried to find the least intrusive way to add the feature of custom list prefixes for AttributedStringVisitor. I've decided to go for ListItemPrefixgenerator changes, as fitting the feature in the styler was not possible.

The PR is not changing the current behavior, but opening a bit the API so one can provide another implementation. See below for an example:

Definition of a custom ListPrefixGenerator:

public class CustomListItemPrefixGeneratorBuilder: ListItemPrefixGeneratorBuilder {
    public func build(listType: List.ListType, numberOfItems: Int, nestDepth: Int) -> ListItemPrefixGenerator {
        return CustomListItemPrefixGenerator(listType: listType, numberOfItems: numberOfItems, nestDepth: nestDepth)
    }
    
}

public class CustomListItemPrefixGenerator: ListItemPrefixGenerator {

    private var prefixes: IndexingIterator<[String]>

    required public init(listType: List.ListType, numberOfItems: Int, nestDepth: Int) {

        switch listType {
        case .bullet:
            let prefix = CustomListItemPrefixGenerator.unorderedPrefix(nestDepth: nestDepth)
            prefixes = [String](repeating: prefix, count: numberOfItems)
                .makeIterator()

        case .ordered(let start):
            prefixes = (start..<(start + numberOfItems))
                .map { CustomListItemPrefixGenerator.orderedPrefix(value: $0, nestDepth: nestDepth) }
                .makeIterator()
        }
    }

    public func next() -> String? {
        prefixes.next()
    }

    private static func unorderedPrefix(nestDepth: Int) -> String {
        switch nestDepth {
        case 0:
            return "●"
        case 1:
            return "○"
        case _ where nestDepth >= 2:
            return "▪︎"
        default:
            return "●"
        }
    }

    private static func orderedPrefix(value: Int, nestDepth: Int) -> String {
        switch value {
        case 1:
            return "1️⃣"
        case 2:
            return "2️⃣"
        case 3:
            return "3️⃣"
        default:
            return "🔢"
        }
    }
}

Usage:

let document = try! markdown.toDocument(.default)
let visitor = AttributedStringVisitor(
    styler: DownStyler(),
    options: .default,
    listPrefixGeneratorBuilder: CustomListItemPrefixGeneratorBuilder()
)
subview.attributedText = document.accept(visitor)

Render:

• Allow customisation of quote paragraph style so we can use custom line heights and spacings.
  • overriding most of the attributes which just effect the styling of the text.
• Make BlockBackgroundColorAttribute public so we can add the attribute in overrides of DownStyler. Use case being when adding syntax highlighting I want to override attributes of the font color and still use the DownLayoutManager to style the background colour

Happy to explain the use cases further

The issue that I’m looking at here is somewhat subtle. I’m mostly looking to start a conversation, at this point. I’m not 100% sure the right way for Down.framework to approach this, so I’m presenting one possible method.

The problem

There’s a lot of issues about fonts with NSAttributedString (#14, #15, #18, #21), so clearly there’s some frustration here.

The way Down.framework works right now is: toAttributedString() calls toHTML(), and then it makes an NSAttributedString from that. This is an easy way to get from a valid Down object to a valid NSAttributedString object, and by Down’s policy of “it’s just a converter”, this is a solution.

The problem is with how NSAttributedString’s interface works. Its default font (as documented in the API) is Helvetica 12, and so that's what you get if you make an NSAttributedString(string: “hello, world”).

But if you use the NSAttributedString(html:Data) initializer, this kicks it into some kind of special HTML rendering mode, which doesn’t use the default font. It uses a tiny serif font. Perhaps it’s trying to emulate Safari, which uses Times by default.

Why Down.framework should care

I can see 3 reasons why Down.framework should get involved here.

First, Markdown is as much a plain text format as an HTML format. People write it like plain text, and it’s designed to be perfectly readable that way. (The existence of Down.framework reinforces this: it renders into 6 different formats, not just HTML.) When people call .toAttributedString(), they aren’t asking for “an NSAttributedString of a Safari view of this Markdown”. They want “an NSAttributedString of this Markdown”.

Second, for every other renderer, Down (like cmark) uses the default font of that backend. The LaTeX output isn’t “LaTeX version of a Safari rendering of Markdown”. It uses the default LaTeX font.

Third, it’s not so easy to change this from outside the library (the previous suggested approach). NSAttributedString is immutable. It’s possible to make a mutable copy and update the fonts, but (AFAICT) it’s really a lot of work. (Personally, I’d write my own Markdown renderer before I went down that path.) Fixing it from inside DownAttributedStringRenderable is easy. Why bother providing an NSAttributedString renderer, if users are going to have to rewrite it themselves to make one that works the way they want?

One solution

My proposed solution is straightforward. Keep using the HTML data path, but prepend a tiny stylesheet which makes it use the default NSAttributedString font. (Menlo, for monospaced text, isn’t a documented default of NSAttributedString, but it’s the standard monospaced font on macOS, and it’s a built-in font on every platform that Down supports.)

This PR makes the stylesheet an optional string, so it’s easy to get the previous Safari-like behavior, too, if you want that:

• To use NSAttributedString’s default font, call .toAttributedString()
• To use the Safari-style rendering, call .toAttributedString(stylesheet: "")
• To use some other custom styles, call .toAttributedString(stylesheet: whateverYouWant)

I’m not entirely happy with this solution, but it’s simple, and provides the default font by default, and allows users to customize it if they need to.

Please help prevent duplicate issues before submitting a new one:

• [x] I've searched other open/closed issues for duplicates before opening up this new issue.

Report

What did you do?

Convert markdown text to attributed string, with bulleted lists spanning multiple lines.

• 123
• 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456
• 789

What did you expect to happen?

The bulleted list to be indented properly.

What happened instead?

The bulleted list is not indented properly.

Please help prevent duplicate issues before submitting a new one:

• [x] I've searched other open/closed issues for duplicates before opening up this new issue.

Report

What did you do?

I am trying to show markdown text using .toAttributedString() to UILabel

What did you expect to happen?

Show makrdown text with images

What happened instead?

Images is in the markdown is replaced with default placeholder image.

Please help prevent duplicate issues before submitting a new one:

• [ x] I've searched other open/closed issues for duplicates before opening up this new issue.

Report

Our SAST report picked up a high vulnerability within this library

• "Potentially unsafe regular expressions. It may take a very long time to run."

What did you do?

Configured SAST to run within GitLab pipelines for our iOS project.

What did you expect to happen?

No high vulnerabilities

What happened instead?

Three high vulnerabilities related to this library has been flagged, this particular one is in the highlight.js file. We're on the latest version of this library and need to be able to reduce all critical and high vulnerabilities in order to ensure we're releasing secure products.

Please help prevent duplicate issues before submitting a new one:

• [ x] I've searched other open/closed issues for duplicates before opening up this new issue.

Report

Our SAST report picked up a high vulnerability within this library

• "If format strings can be influenced by an attacker, they can be exploited, and note that sprintf variations do not always \0-terminate (CWE-134)."

What did you do?

Configured SAST to run within GitLab pipelines for our iOS project.

What did you expect to happen?

No high vulnerabilities

What happened instead?

Three high vulnerabilities related to this library has been flagged, two of which are in the cmark/config.h file. We're on the latest version of this library and need to be able to reduce all critical and high vulnerabilities in order to ensure we're releasing secure products.

Please help prevent duplicate issues before submitting a new one:

• [ x] I've searched other open/closed issues for duplicates before opening up this new issue.

Report

Our SAST report picked up a critical vulnerability within this library

• "Bracket object notation with user input is present, this might allow an attacker to access all properties of the object and even it's prototype, leading to possible code execution."

What did you do?

Configured SAST to run within GitLab pipelines for our iOS project.

What did you expect to happen?

No critical vulnerabilities

What happened instead?

One critical vulnerability related to this library has been flagged in the highlight.js file. We're on the latest version of this library and need to be able to reduce all critical and high vulnerabilities in order to ensure we're releasing secure products.