• [x] Only write the metadata if its set and it is not equal to any existing metadata for the index.
• [x] Remove any stored metadata if the property is .None.
• [ ] Test coverage

These changes are minor, opinion-based reorganizations with no functional change; thus this PR may be closed with no hard feelings if this code churn is undesirable.

• https://github.com/danthorpe/YapDatabaseExtensions/commit/d21165b3479f4abd7ef891dc113f83de05cf6459 separates YapDB.Index's Hashable and CustomStringConvertible implementations; they're not really related and it took me a moment to realize there wasn't a reason for them to be grouped together like that.

• https://github.com/danthorpe/YapDatabaseExtensions/commit/8df385e2d9f69c98480946253c03802aaf1b5699 groups free functions operating on Persistables alongside the Persistable protocol definition, as so:

  

Hey Dan

Thanks for the awesome work on the extensions! Excited about moving allot of my data layer over to structs.

I have a couple of questions if you have the time?

Reading

I have a User struct, see below for its implementation, it adopts the Persistable protocol.

How do I best go about cleanly reading records by key? I thought something like this would do the trick but seems to complain about ambiguous reference to member.

let user: User? = User.readByKey("1")

Updating

This is more of a general question as how you approach updating records in your database, do you normally pull out an existing record, create a new struct based on that and then write the new struct to the database with the existing key?

Metadata

I could not seem to find a way to set the default Void metadata on the user struct. This seems wrong?

var metadata: NSNumber? {
    return .None
}

User

import Foundation
import ValueCoding
import YapDatabaseExtensions

struct User: Equatable {

    // MARK: - Internal Properties

    let ID: Int

    let name: String

    // MARK: - Private Properties

    // MARK: - Initializers

    init(ID: Int, name: String) {
        self.ID = ID
        self.name = name
    }

    // MARK: - Internal Methods

    // MARK: - Private Methods
}

extension User: ValueCoding {

    // MARK: - ValueCoding

    typealias Coder = UserCoder
}

extension User: Persistable {

    var identifier: Identifier {
        return "\(ID)"
    }

    static var collection: String {
        return "Users"
    }

    var metadata: NSNumber? {
        return .None
    }
}

func == (lhs: User, rhs: User) -> Bool {
    return lhs.ID == rhs.ID
}

I'm making some changes to the read and write APIs, following on from my first attempt in #39. They are as follows:

There are 6 different kinds of "types" which are supported, as before, for clarification these are:

| Description | Encoding Style | Metadata Encoding Style | | --- | --- | --- | | Object with no metadata | NSCoding | N/A | | Object with Object metadata | NSCoding | NSCoding | | Object with Value metadata | NSCoding | Saveable | | Value with no metadata | Saveable | N/A | | Value with Object metadata | Saveable | NSCoding | | Value with Value metadata | Saveable | Saveable |

The APIs are only available when your types correctly implement one of these 6 patterns, and the changes being introduced which uses constrained protocol extensions make writing these APIs far less error prone as I only need to write the where constraint once for each combination.

So, here's what they look like, I'm gonna use Foo as my Persistable type, and you should assume that it could be any of the above 6 patterns.

Reading

If you have a YapDatabaseConnection...

connection.read { transaction in

    // Get an optional Foo by YapDB.Index
    if let foo = Foo.read(transaction).atIndex(anIndex) {
        // etc
    }

    // Get an optional Foo by key (String)
    if let foo = Foo.read(transaction).byKey(aKey) {
        // etc
    }

    // this is [Foo], could be empty, count could be less than someIndexes
    let foos = Foo.read(transaction).atIndexes(someIndexes)

    // this is [Foo], could be empty, and count could be less than someKeys
    let foos2 = Foo.read(transaction).atKeys(someKeys)

    // Get all the Foo items in the db, as [Foo], could be empty.
    let foos3 = Foo.read(transaction).all()

    // Get the [Foo] objects which exist for the given keys, and return the [String] keys which are missing.
    let (foos4, missingKeys) = Foo.read(transaction).filterExisting(someKeys)
}

But you can also perform the same functionality directly with a connection too.

if let foo = Foo.read(connection).atIndex(anIndex) {
    // etc
}
if let foo = Foo.read(connection).byKey(aKey) {
    // etc
}
let foos = Foo.read(connection).atIndexes(someIndexes)
let foos2 = Foo.read(connection).atKeys(someKeys)
let foos3 = Foo.read(connection).all()
let (foos4, missingKeys) = Foo.read(connection).filterExisting(someKeys)

The same APIs but with YapDatabase instances now exist only internally to prevent poor programming practices. They are used internally for unit testing.

Writing

If you have a YapDatabaseConnection...

let foo = Foo() 
let foos = Set([Foo(), Foo(), Foo()])

connection.write { transaction in 
    // write one item at a time
    foo.write.on(transaction)
    // write a SequenceType of items
    foos.write.on(transaction)
}

Alternatively, to perform writes in the same transaction, but directly on a connection, you can do....

// Write synchronously using the connection.
foo.write.sync(connection)
foos.write.sync(connection)

// Write asynchronously using the connection.
foo.write.async(connection) { print("finished writing") }
foos.write.async(connection) { print("finished writing") }

// Return an NSOperation which will perform the write using the connection
queue.addOperation(foo.write.operation(connection))
queue.addOperation(foos.write.operation(connection))

this is handy if you only have a single item or single array of items to write - which is probably the most common scenario.

Note also that these APIs do not have return values. Previously the write() function would return the item being written, however this is fairly redundant and would have made the implementation impossible/much harder :).

Framework consumers cannot write directly to YapDatabase instances now, although internally there is a .to(database) API for unit testing.

But what about metadata?

This PR will deprecate ObjectMetadataPersistable and ValueMetadataPersistable which are both replaced by MetadataPersistable which is:

public protocol MetadataPersistable: Persistable {
    typealias MetadataType
    var metadata: MetadataType? { get set }
}

For types which implement MetadataPersistable, when writing, YapDatabaseExtensions will write the metadata property (if it exists) to YapDatabase. When reading, YapDatabaseExtensions will set the property (if it exists in the database) before returning the item. This is the best and ideal behavior, as you get optional metadata types and no additional storage costs. Although there are no constraints on MetadataType in the protocol, it must implement either NSCoding or Saveable for the extensions to perform as expected.

Original issue discussions

Looking for some input here from uses who have metadata types associated with their models. How do you define your models? Do you save the metadata inside the model in addition to having YapDatabase save it as metadata? If not, how would you like to read your models and/or metadata?

This is how I've previously used YapDatabase's metadata functionality: (which might not be what you assumed).

struct Foo: ValueMetadataPersistable {

    struct Metadata {
          let created: NSDate
          let modified: NSDate?
    }

    let metadata: Metadata
}  

1. the MetadataType is a nested type within my model type .i.e. Foo.Metadata.
2. the metadata is a property of the model - which means I would archive it inside the model object - meaning that there is no need to "read" it in addition to reading the model itself.

The reason for this approach is that, I think this is how YapDatabase is designed to work - when you read objects for key, you get back the object, you can retrieve the metadata separately using another API. The purpose is to provide quicker (separate caches & presumably less decoding effort) access to the metadata of objects in extensions such as YapDatabase Views etc. When you write models to the database, the object and metadata is written in the same call. So, in other words the cost of having faster access to that metadata is that YapDatabase will effectively store it twice, once as part of the model and then again in a metadata only blob.

If you follow the above approach, there is no explicit need to read (model & metadata) because the model includes the metadata. However, you may wish to read only the metadata, in which case try readMetadataAtIndex: functions. Although I can add in the PR other functions like readMetadata(key: String) functions.

Okay, so after discussions below, I'm gonna add functionality for "external metadata" which means that it is only stored inside YapDatabase - potentially, renaming/replacing the current metadata based protocols

protocol ValueMetadata: Persistable {
    typealias MetadataType: Saveable

   var metadata: MetadataType? { get set }
}

with updates to read and write functions which will automatically store metadata (if set) during a write, and will set the property (if available) when reading.

Currently I can write my Item without writing its Metadata by setting the metadata property to nil, there's no way to conversely write the Metadata without also writing the Item.

There also doesn't appear to be a way to read either one without reading them both. This may seem odd, given that my original request was that I wanted to be able to read them together, which I now can, but I'd still like to be able to read them separately if needed.

Neither of these are really that big of a deal and may just be me prematurely optimizing, but as I'm refactoring my code to use the new API these have come up and I thought I'd run them by you.

While reviewing the code, I found a few places where the inline documentation could be clearer; and I also wanted to note the getting-started steps with Carthage in case other devs who check out this repo aren't already familiar.

The only problem with adding functions directly to YapDatabaseReadWriteTransaction (etc) is that every function needs to be generic. This can be a bit of a problem as this is the generic where clause for value types with value type metadata..

extension Readable
    where
    ItemType: Saveable,
    ItemType: MetadataPersistable,
    ItemType.ArchiverType: NSCoding,
    ItemType.ArchiverType.ValueType == ItemType,
    ItemType.MetadataType: Saveable,
    ItemType.MetadataType.ArchiverType: NSCoding,
    ItemType.MetadataType.ArchiverType.ValueType == ItemType.MetadataType {

which is very verbose and error prone - although less so now that the protocol types are relatively stable.

So, there are a couple of options which might work as alternatives.

1. Similar to the Read and Write structures, we could wrap YapDatabaseReadWriteTransaction, YapDatabaseConnection inside a generic type, which then has generic methods for read & write. e.g.

   if let person: Person = transaction.read.byKey(key) {
      // etc
   }
   

   which is probably the best, although not sure that this is possible.

2. ~~Add a generic ItemType to the facade protocols ReadTransactionType etc and then define the functions in protocol extensions?~~ This will work - but don't think that it'll be possible to have the YapDatabase types then conform to the protocols. - correct, not a viable solution.

I think it's going to have to be fully generic functions. But the can be added to the facade protocols which is better for testing at least. Anyway, so the APIs are as follows...

let item: Item? = transaction.readAtIndex(index)
let item: Item? = transaction.readByKey(key)
let items: [Item] = transaction.readAtIndexes(indexes)
let items: [Item] = transaction.readByKeys(keys)

let item: Item? = connection.readAtIndex(index)
let item: Item? = connection.readByKey(key)
let items: [Item] = connection.readAtIndexes(indexes)
let items: [Item] = connection.readByKeys(keys)

transaction.write(item)
transaction.write(items)
connection.write(item)
connection.write(items)
connection.asyncWrite(item) { print("did write") }
connection.asyncWrite(items) { print("did write") }

transaction.remove(item)
transaction.remove(items)
connection.remove(item)
connection.remove(items)
connection.asyncRemove(item) { print("did remove") }
connection.asyncRemove(items) { print("did remove") }

where Item is one of the following type patterns:

• [ ] NSCoding class with no metadata
• [x] NSCoding class with NSCoding class metadata
• [x] NSCoding class with Saveable value metadata
• [ ] Saveable value with no metadata
• [x] Saveable value with NSCoding class metadata
• [x] Saveable value with Saveable value metadata

Notes:

1. Not going to provide a functional transaction.readAll() -> Use the Persistable extension for this.

This pull requests corrects the spelling of CocoaPods 🤓 https://github.com/CocoaPods/shared_resources/tree/master/media

One day I’ll make a bot that looks through the READMEs of all Pods, looks to see if it uses “Cocoapods” and PRs “CocoaPods” :D

— Ørta (@orta) February 10, 2016

Created with cocoapods-readme.

When reading the README, I found it strange that the paragraph

There are also two styles of API. The functional API works on YapDatabase types, …

was under the “Correct Type Patterns” header.

This minor change resolves that confusion, IMO, by introducing a new header above that paragraph and bumping the Functional and Persistable API headers down to h3 from h2.

YapDatabaseExtensions.swift was where I found a couple, but thought it'd be a good idea to flag it in an issue and do a search through the rest of the files when someone gets a chance.

Hi! Great work abstracting YapDB!

I'm wondering if using newConnection() for all exposed API calls is performant, as YapDatabase recommends to keep connections around for caching reasons.

I generally cache the connections in properties from where I'm accessing YapDB; Usually one for readonly operations which will have sync readonly calls, another one for readwrite operations which will have asynchronous calls.

Have you tried this in a big application and experienced any issues?

I'm using Xcode 8.0/Swift3

I used Cocoapods to install YapDatabaseExtensions, along with it ValueCoding was installed. The ValueCoding.swift has 12 or so compiler errors. Most are complaining about >> Use of undeclared type 'Generator' and Value of type 'S' has no member 'flatMap'

How do I fix this? Thank you

Hi Dan,

You mentioned a Swift 3 port was in progress a while ago. My team's Swift 3 port is currently blocked by YapDatabaseExtensions, so if there's any way I can help with the this port, please let me know.

Dan, thanks for making YapDatabaseExtensions!

This PR contains some sutble but signficant changes I consider important for improving the performance and flexibility of this framework.

Metadata changes

This PR improves the handling metadata by YapDatabaseExtensions. I see there have been some previous attempts at reforming the metadata handling but none had been merged yet. I hope this solution is found to be acceptable.

The var metadata property and associatedtype MetadataType as been removed from the Persistable protocol.

YapDatabase does not place a constraint on the types of metadata that can be associated with values, and this is a useful feature. Having a single MetadataType per Persistable type reduced this flexibility.

An important performance design aspect of metadata in YapDatabase is the ability able to read and write it independently of the value. Unfortunately, with the current design, when you have a Persistable with a MetadataType, all reads will do an extra read for the metadata and then set the metadata on the value. This means twice the number of reads (even when the metadata isn't needed after the read) and creating a copy of the value when value types are used, as value types are immutable. If one used objects for their Yap values they'd have to make them mutable which isn't compatible with Yap's sharing policy. In practice, values and metadata do tend to be written together, but for reads the opposite is true.

It is still possible to read/write a value and a metadata at the same time - the methods in the "With[Object|Value]Metadata" files have been renamed to have "with metadata" in the name. For example, write is now writeWithMetadata. These "with metadata" methods take and return a YapItem<Value, Metadata>, which is a simple new struct akin to a tuple, containing the value, metadata and their associated type information since it's generic. This is preferable to a tuple because anonymous tuple types cannot be extended, while structs obviously can. By using this type-rich YapItem, the desirable type-safety features in YapDatabaseExtensions are not impacted.

Additional changes

Additionally, the methods that query for multiple YapDB.Index (such as readAtIndexes) no longer flatMap the return value, so it is now easy to determine which YapDB.Index were not found in the database. If one doesn't need this information they can easily flatMap the return value.

How to test

All tests have been updated and pass.

I merged danthorpe:development into my branch just before submitting this PR. It merged with no conflicts.

This is definitely a question, not a bug report, so read accordingly :)

I've recently been re-structuring my data a bit so that I have the fields that I sort on in the Metadata and everything else in the object data. Reading https://github.com/yapstudios/YapDatabase/wiki/Views#initialization-tips suggests that is the recommended way to avoid unnecessary recalculation of the sortings. The problem I'm running into is that because YapDatabaseExtensions writes the metadata every time it writes an object, even if that metadata hasn't changed, every time I write an object the database my sort blocks are being run, even though I haven't updated their sort-able properties.

My question is, am I missing something? Is there a way that I can accomplish this with YapDatabaseExtensions or is this an approach that it's just not designed for?