YapDB is a collection/key/value store with a plugin architecture. It's built atop sqlite, for Swift & objective-c developers.

Hi,

Would it be possible to use SQLCipher (http://www.sqlcipher.net) instead of the built-in SQLite engine? I don't see why not, but I'm just asking for clarification.

I wanted to use Yap to create a sort of persistent work queue.

The example I'm thinking of is a messaging app. Each message moves through multiple 'states' like being encrypted, sending to the server, and handling error cases.

So in this case the 'encryption listener' would be on the end of queue for messages needing encryption. Then once encrypted they would be stored back into yap and marked for needing sending. And then the 'sending listener' would pick up the message and send it over the wire. This helps us in the case where the device is offline so we don't loose any messages or are unsure of the messages state.

My question is what's the best yap tool for the job? It seems Views would be the closest even if they have a few other features that wouldn't be necessary. Or would it be worthwhile to create a new extension for this?

Here's a stab at the dynamic groups discussed in #34. I don't think this pull request is ready, I haven't tested it manually or written unit tests. Will work on those more tonight.

This is a first version to allow YapDatabase to work across multiple processes, you have to set the enableMultiprocessSupport option when creating the database. You can add a CrossProcessNotifier extension to be notified when another process has modified the database.

Allowing the database to work across process requires us to disable a few optimizations when the user has enabled multiprocess support. The optimizations are still available when in single process mode

@robbiehanson here is a summary of changes, would be glad if you could comment on those

Summary of changes

• added a busy_handler when creating the database to make sure we can open several databases in multiprocess mode (https://github.com/yapstudios/YapDatabase/pull/283/files#diff-c9a9379c637725775619d9e0b7933ca7R75)
• added a enableMultiprocessSupport option to database (https://github.com/yapstudios/YapDatabase/pull/283/files#diff-2e3be8a8d3931c503fa36d74f2a407e9R273)
• we read the last snapshot from the database instead, instead of restarting from 0 (https://github.com/yapstudios/YapDatabase/pull/283/files#diff-c9a9379c637725775619d9e0b7933ca7R1350)
• replaced busy_timeout with a busy_handler which loops and sleep 50ms, outputs a log each 4 cycles if the lock was not acquired (https://github.com/yapstudios/YapDatabase/pull/283/files#diff-4fc49c6d91fb2d5322f8ee0387c2f447R72)
• updated preReadTransaction (https://github.com/yapstudios/YapDatabase/pull/283/files#diff-4fc49c6d91fb2d5322f8ee0387c2f447R2157)
  • added a flag || isMultiprocess to force reading the latest snapshot number from the database file (instead of the last version from the singleton YapDatabase object)
  • when we get the changeset, if we miss one snapshot number in the list of changesets, we return nil instead of the array of changeset, and we clear the cache of the connection and all its extensions
  • we do this by counting the number of relevantChangesets which should match the increase in the snapshot number maxSnapshot - connectionSnapshot
  • after flushing memory, we still update the snapshot of the current transaction to the one that we read from the database
  • then do we also need to update the global [database snapshot] version? when should we update this one? there might be a case where the current connection was just updated from the database file, so it is > to the one in [database snapshot]
• updated preReadWriteTransaction (https://github.com/yapstudios/YapDatabase/pull/283/files#diff-4fc49c6d91fb2d5322f8ee0387c2f447R2442)
  • replaced beginTransaction with beginImmediateTransaction to acquire the lock if we are in multiprocess mode, so there is no concurrent readTransaction while doing a writeTransaction, which would cause issues (allowing read while there is a write transaction would require to extend the race-condition check to the multiprocess case, which would be complex)
  • in multiprocess mode, we force reading the current snapshot from the database file, and we update with changesets or clean cache if one is missing
• updated postReadWriteTransaction
  • when a readWrite connection makes changes and publishes a changeset, if it detected that the database was externally modified, it sends a changeset with the YapDatabaseModifiedExternallyKey set to TRUE to notify other connections and extensions that they should clean their cache
• added a CrossProcessNotification extension that allows to receive a YapDatabaseModifiedExternallyNotification when another process updates the database (it uses an identifier so you can have several databases with different identifiers if needed), but this does not transmit the changeset (https://github.com/yapstudios/YapDatabase/pull/283/files#diff-0fa70ef80dace0725d7786e5ac3cb17aR20)
• YapDatabaseModifiedNotification: right now we send changesets cross-process, so we don't send this modification when the database has been externally modified
• when a changeset with the YapDatabaseModifiedExternallyKey is sent, methods like hasChangeForCollection, hasChangeForKey, all respond TRUE
• when updating from a changeset (YapDatabaseConnection.noteCommitedChangeset), if changeset.snapshot != connection.snapshot + 1, we flush the cache, then update the changeset
• QUESTION: https://github.com/yapstudios/YapDatabase/pull/283/files#diff-4fc49c6d91fb2d5322f8ee0387c2f447R327
• QUESTION: what is the use of YPDBLogTrace and should we remove this line https://github.com/yapstudios/YapDatabase/pull/283/files#diff-cdbb67ebc7a257151869cc4413f1dc9aR146

You can see an example https://github.com/yapstudios/YapDatabase/pull/283/files#diff-4a13c0a0aa202a99f4ebe0ab5d3e6175R20

NOTE: right now the ModifiedExternally notification works by sending the pid as payload in libnotify, so if the current process has the same pid, it does not send the notification, and only other processes send it. Another way to go about it would be to send the snapshot number as payload, and each process check whether its [database snapshot] has the same value, otherwise it sends a ModifiedExternally notification (so for instance if the database instance had the time to update before receiving the libnotify message, we don't have to send another ModifiedExternally notification since the database was already updated)

Hi, I'm in the process of migrating an app over from core data to YapDatabase. I'm currently using Views to replace my NSFetchedResultsController and I've run into a snag. I have a list of events that are grouped by day. So I need a section, or group, for each day for which there is an event. The problem is that mappings require you to know beforehand the names of all the groups. Is there a way to do this using views and mappings?

Thanks.

It would be great if YapDatabase did not force me to include CocoaLumberjack. It may be useful for the project itself but I do not wish to include it in my applications.

Hi Robbie and others,

We're seeing a very odd SIGSEGV crash that makes up about 90% of our app's crashes right now. The stack trace looks like this:

0 CoreFoundation 0x0000000183a813d4 -[__NSArrayM dealloc] + 140 1 CoreFoundation 0x0000000183a82784 -[__NSDictionaryM dealloc] + 148 2 libobjc.A.dylib 0x0000000183202b54 object_cxxDestructFromClass(objc_object_, objc_class_) + 144 3 libobjc.A.dylib 0x000000018320e080 objc_destructInstance + 88 4 libobjc.A.dylib 0x000000018320e0e0 object_dispose + 24 5 YapDatabase 0x00000001027ba65c -YapDatabaseViewConnection .cxx_destruct 6 libobjc.A.dylib 0x0000000183202b54 object_cxxDestructFromClass(objc_object_, objc_class_) + 144 7 libobjc.A.dylib 0x000000018320e080 objc_destructInstance + 88 8 libobjc.A.dylib 0x000000018320e0e0 object_dispose + 24 9 YapDatabase 0x00000001027b6c40 -YapDatabaseViewConnection dealloc 10 CoreFoundation 0x0000000183ac7984 -[__NSDictionaryM removeAllObjects] + 508 11 YapDatabase 0x000000010273d528 -YapDatabaseConnection dealloc 12 CoreFoundation 0x0000000183ae51ac -[NSSharedKeyDictionary dealloc] + 84 13 Foundation 0x000000018446aee4 -[NSConcreteNotification dealloc] + 84

The only likely candidate we've found for that NSMutableDictionary instance that lines up with the stack trace is group_pagesMetadata_dict, which contains arrays of YapDatabaseViewPageMetadata objects. Our current line of thinking is that a deallocation of the YapDatabaseViewState ivar of YapDatabaseViewConnection deallocates the dictionary instance, and then an object under a pointer in one of the arrays is being freed. Very weird considering all of this is ARC managed, and that removals from the YapDatabaseViewPageMetadata arrays are controlled through the API.

Any thoughts on where to look next would be most appreciated. We've so far been unable to identify a place where a non-arc-like release might occur. It's also worth noting that our interactions with YapDatabase's API are entirely from Swift - possibly leaving the door open for things like group names (Strings) to have lingering swifty memory management

Hi,

If you needed to do fast searching for specific values, not just a key lookup, how would you accomplish that? For example, doing a substring search for "Term" within a Movies database and having it return a list of movies with the Movie Titles that contain the term "Term".

For example:

Movie Title:

The Terminator Terminator 2: Judgement Day The Terminal

Right now I'm using SQLCipher with the FTS3 full text search engine in SQLite. But there's an overhead in maintaining the separate FTS search engine that I'd like to remove.

I'm assuming that your system would require me to fetch all data into memory and then do an in-memory filter using something like an NSPredicate. There would be a big memory and performance impact in doing so I would think.

when i using carthage, there was no macos scheme in YapDatabase.xcodeproj, can you add it to your todo list, please? (sorry. my english, perhaps, is coarse)

In my app, I implemented a sync with Dropbox. In restoring the database files from a Dropbox backup, I replace the database files (.sqlite). Then I run this code:

YapDatabaseConnection *logsDatabaseConnection = [_logsDatabase newConnection];
[logsDatabaseConnection flushMemoryWithFlags:YapDatabaseConnectionFlushMemoryFlags_All];

YapDatabaseConnection *songsDatabaseConnection = [_songsDatabase newConnection];
[songsDatabaseConnection flushMemoryWithFlags:YapDatabaseConnectionFlushMemoryFlags_All];

_repertoiresDatabase = [[YapDatabase alloc] initWithPath:_repertoiresDatabasePath];
_songsDatabase = [[YapDatabase alloc] initWithPath:_songsDatabasePath];
_logsDatabase = [[YapDatabase alloc] initWithPath:_logsDatabasePath];

Even after this, I have to restart the app for transactions to have the new database. How can I fix this?

Hi,

We are using YapDatabase with indexes, views and external pictures. Now there is a need to implement support for full database backups which can be restored later.

I am planning to “export” all known objects into the new clean YapDatabase excluding any additional view tables and metadata for objects. In this case, I believe, there is no need to keep SHM and WAL files around.

So is there any safe way to disable SHM and WAL per-database?

Thanks in advance! Vadim

I'm theorising my solution for implementing Full Text Search, and I had a few questions to ensure my understanding of the module is correct.

Full Disclosure; I have not tried writing or experimenting with any code.

Models

I have two distinct model objects which I need to index for using with YapDB's FTS module.

struct Foo {
    let identifier: String
    let title: String
    let keywords: [String]
}

struct Bar {
    let identifier: String
    let title: String
    let author: String   
    let summary: String?
}

A distinct merged struct can be created from the above like so:

enum SearchableType {
    case Foo
    case Bar
}

struct Searchable {
    let identifier: String
    let title: String
    let keywords: [String]?
    let author: String?
    let summary: String?
    let type: SearchableType
}

View

For my search view, I need to show both matching types (Foo and Bar) based on the user's input.

Question

As per the documentation for FTS, the YapDatabaseFullTextSearch initialiser takes a columns argument with propertiesToIndex as its value.

In my case, title and identifier are the only common property across the two models. What the documentation is unclear about are conditional values for the columns.

A: Assuming that conditional values are unsupported (i.e. the FTS module would not like NULL values in these columns), should I create two separate handlers and YapDatabaseFullTextSearch extensions for each struct (and forego the merged struct)?

B: If conditional values are supported, do I need to take additional care when writing the queries for fetching results? Repurposing one of the examples from the documentation:

// struct.author matches: john
// struct.summary contains phrase: "board meeting"

query = "author:john summary:\"board meeting\""
ftsTransaction.enumerateKeysAndObjects(matching: query) { (collection, key, obj, stop) in
    // ...
}

Does the FTS module take care of only looking at non-NULL values in its table? Or do I need to modify the above query to explicitly tell it to skip over NULL values?

TL;DR YapDatabaseViewSortingBlock declared as returning nonnull objects but in fact can return nil in some cases

Details in this comment https://github.com/yapstudios/YapDatabase/issues/459#issuecomment-812550132

"hello world" code can run, but when I terminal my app and just run the code below, str is nil

let db = YapDatabase() db.registerCodableSerialization(String.self, forCollection: "test")

let connection = db.newConnection()

//connection.readWrite {(transaction) in // transaction.setObject("hello", forKey: "world", inCollection: "test") //}

connection.read {(transaction) in let str = transaction.object(forKey: "world", inCollection: "test") as? String }

I'm having an issue where relationships are causing the wrong object to be deleted. Unfortunately this is in a very complex code path, so I can't give any steps to reproduce the behaviour, but I can report my observations. While I can't guarantee that there are no bugs in our code, the behaviour I'm observing from Yap does seem inconsistent.

This is a pretty serious bug because it can cause seemingly random data deletions in a Yap app that uses relationships.

Mechanism

I tracked this down to [YapDatabaseTransaction removeObjectForCollectionKey:withRowId:] being called with the wrong row ID. I added the following code block to the start of the method and put a breakpoint in it.

    YapCollectionKey *storedKey = [self collectionKeyForRowid:rowid];
    if (![storedKey isEqual:cacheKey]) {
        int64_t correctRowID = 0;
        BOOL result = [self getRowid:&correctRowID forCollectionKey:cacheKey];
        
        NSLog(@"====rssr yap is removing rowid %ld. Caller thinks the key is %@ but it's actually %@. The correct rowid to delete is most likely %ld (fetch rowid result was %@)", rowid, cacheKey, storedKey, correctRowID, result ? @"TRUE": @"FALSE");
    }

When the issue happens I get the following log:

2021-01-14 18:14:12.558989-0500 Daylite[2351:1444308] ====rssr yap is removing rowid 9691. Caller thinks the key is <YapCollectionKey collection(RemoteMutationsReferences) key(72D89780-77D6-436F-BEEE-5024AA600D2F_1)> but it's actually <YapCollectionKey collection(accounts) key(6)>. The correct rowid to delete is most likely 0 (fetch rowid result was FALSE)

So it looks like the wrong row id is stored in the edge (it doesn't match the destination key)

The call to remove the object is in YapDatabaseRelationshipTransaction.m, at the end of "Enumerate all edges where source node is the deleted node" around line 3727:

								[databaseRwTransaction removeObjectForKey:edge->destinationKey
								                             inCollection:edge->destinationCollection
								                                withRowid:edge->destinationRowid];

My Steps To Reproduce

The workflow in our app is so complicated that it isn't possible to glean the real requirements for this issue from them, but I think there is some information about the preconditions.

1. View the messages list (it's backed by a YapDatabaseAutoView).
2. Start swiping to delete messages from the list.
3. Each deletion will trigger a server sync that will push the deletion up to the server and then fetch any new data (There does not need to be any new data to trigger the bug however).
4. After a certain number of messages are deleted one of them will come back. We do have code that will read back a deleted object if we fail to push the deletion to the server, but I haven't verified that this is the cause of the object coming back (due to some of the issues listed below).
5. Swipe to delete the same message again. This second deletion succeeds locally, but when pushed to the server it causes the entire account object to be deleted while cleaning up some unrelated relationships.

Notes & Observations

1. In my debug code above I try to fetch the correct rowID from the CollectionKey and it comes back as 0, so I'm guessing that the intended target of the edge has already been deleted.
2. I don't think that the bug is caused when the edge is firing in step 5, I believe that it's actually in step 4 when the object comes back. If I quit the application between steps 4 and 5 the message that 'comes back' is not there, so it looks like it wasn't fetched back by our application and persisted, it's an in memory issue.
3. If you look at the output of the debug log I put above it tries to resolve the real rowid of the destination, and it comes back as 0. I believe that the 'real' target of the relationship has already been deleted, but the machinery in edge processing doesn't realize this because of the rowid issue. (That's just a guess at this point). I have stepped through the call to fetch the rowid and the zero doesn't come from the cache and it does do a database query to attempt to retrieve the rowid.
4. I have code that performs a rollback in some cases, but I did not see the log from this code at all in some cases where this issue has occurred.
5. The edge that is firing was not created in the same transaction as the one that is deleting everything.
6. While investigaing this we have noticed a few other inconsistencies where some of our objects were in an unexpected state. Broken references between objects that are updated in a single transaction. We weren't able to fully rule out a bug in our code, so I don't know if that issue is related to this one.
7. If the message that 'came back' was caused by our sync code, the new message would have been assigned a new id, since we assign ids to objects sequentially and don't re-use them.
8. We upgraded our version of Yap recently, and commits b3d459964f2fbd55a8ce0e9f59f97f37551a11dc and dda1ecea8d3292facb9db7d989418875ebc07c70 are both new to our repo. I don't believe that we had this problem before, but it is a bit hard to tell if this issue is happening or not.
9. This issue is heavily timing dependant. I added a lot of logs to both our code and to Yap and the issue stopped happening. Clearly this is exposing an issue in some special case for concurrent code that doesn't get hit all the time.
10. I can reproduce this issue with a debug build of Yap.
11. The object that is incorrectly targeted by the edge is the root of our object hierarchy. Not only does this trigger the deletion of everything, I suspect that it is the first row in our database, which may be a clue to the mechanism of operation.

Workaround

As a workaround I'm going to try removing the edge that is firing and deleting the wrong objects. We don't need this particular relationship anymore, so hopefully it will prevent this issue. But that seems like a band-aid solution, since the underlying problem is likely to remain. I'm a bit worried about trusting the relationship extension.

If I call asyncRegisterExtension for storage with not registered serialisation for some persisted object, YapDatabase will throw unprocessed exception.

I want to perform lazy serialisation registration for my objects.