🛶Shallows is a generic abstraction layer over lightweight data storage and persistence.

Last update: Dec 3, 2022

Related tags

Overview

Shallows

Shallows is a generic abstraction layer over lightweight data storage and persistence. It provides a Storage type, instances of which can be easily transformed and composed with each other. It gives you an ability to create highly sophisticated, effective and reliable caching/persistence solutions.

Shallows is deeply inspired by Carlos and this amazing talk by Brandon Kase.

Shallows is a really small, component-based project, so if you need even more controllable solution – build one yourself! Our source code is there to help.

Usage

let kharkiv = City(name: "Kharkiv", foundationYear: 1654) diskStorage.set(kharkiv, forKey: "kharkiv") diskStorage.retrieve(forKey: "kharkiv") { (result) in if let city = try? result.get() { print(city) } } ">

struct City : Codable {
    let name: String
    let foundationYear: Int
}

let diskStorage = DiskStorage.main.folder("cities", in: .cachesDirectory)
    .mapJSONObject(City.self) // Storage

let kharkiv = City(name: "Kharkiv", foundationYear: 1654)
diskStorage.set(kharkiv, forKey: "kharkiv")

diskStorage.retrieve(forKey: "kharkiv") { (result) in
    if let city = try? result.get() { print(city) }
}

Guide

A main type of Shallows is Storage. It's an abstract, type-erased structure which doesn't contain any logic -- it needs to be provided with one. The most basic one is MemoryStorage:

let storage = MemoryStorage<String, Int>().asStorage() // Storage

Storage instances have retrieve and set methods, which are asynhronous and fallible:

storage.retrieve(forKey: "some-key") { (result) in
    switch result {
    case .success(let value):
        print(value)
    case .failure(let error):
        print(error)
    }
}
storage.set(10, forKey: "some-key") { (result) in
    switch result {
    case .success:
        print("Value set!")
    case .failure(let error):
        print(error)
    }
}

Transforms

Keys and values can be mapped:

let images = storage
.mapValues(to: UIImage.self,
transformIn: { data in try UIImage.init(data: data).unwrap() },
transformOut: { image in try UIImagePNGRepresentation(image).unwrap() }) // Storage
enum ImageKeys : String {
case kitten, puppy, fish
}
let keyedImages = images
.usingStringKeys()
.mapKeys(toRawRepresentableType: ImageKeys.self) // Storage
keyedImages.retrieve(forKey: .kitten, completion: { result in /* .. */ })
">
let storage = DiskStorage.main.folder("images", in: .cachesDirectory) // Storage
let images = storage
    .mapValues(to: UIImage.self,
               transformIn: { data in try UIImage.init(data: data).unwrap() },
               transformOut: { image in try UIImagePNGRepresentation(image).unwrap() }) // Storage

enum ImageKeys : String {
    case kitten, puppy, fish
}

let keyedImages = images
    .usingStringKeys()
    .mapKeys(toRawRepresentableType: ImageKeys.self) // Storage

keyedImages.retrieve(forKey: .kitten, completion: { result in /* .. */ })

NOTE: There are several convenience methods defined on Storage with value of Data: .mapString(withEncoding:), .mapJSON(), .mapJSONDictionary(), .mapJSONObject(_:) .mapPlist(format:), .mapPlistDictionary(format:), .mapPlistObject(_:).

Storages composition

Another core concept of Shallows is composition. Hitting a disk every time you request an image can be slow and inefficient. Instead, you can compose MemoryStorage and FileSystemStorage:

let efficient = MemoryStorage<Filename, UIImage>().combined(with: imageStorage)

It does several things:

When trying to retrieve an image, the memory storage first will be checked first, and if it doesn't contain a value, the request will be made to disk storage.
If disk storage stores a value, it will be pulled to memory storage and returned to a user.
When setting an image, it will be set both to memory and disk storage.

Read-only storage

If you don't want to expose writing to your storage, you can make it a read-only storage:

let readOnly = storage.asReadOnlyStorage() // ReadOnlyStorage

Read-only storages can also be mapped and composed:

()
.backed(by: immutableFileStorage)
.asReadOnlyStorage() // ReadOnlyStorage
">
let immutableFileStorage = DiskStorage.main.folder("immutable", in: .applicationSupportDirectory)
    .mapString(withEncoding: .utf8)
    .asReadOnlyStorage()
let storage = MemoryStorage<Filename, String>()
    .backed(by: immutableFileStorage)
    .asReadOnlyStorage() // ReadOnlyStorage

Write-only storage

In similar way, write-only storage is also available:

let writeOnly = storage.asWriteOnlyStorage() // WriteOnlyStorage

Different ways of composition

Compositions available for Storage:

.combined(with:) (see Storages composition)
.backed(by:) will work the same as combined(with:), but it will not push the value to the back storage
.pushing(to:) will not retrieve the value from the back storage, but will push to it on set

Compositions available for ReadOnlyStorage:

.backed(by:)

Compositions available for WriteOnlyStorage:

.pushing(to:)

Single element storage

You can have a storage with keys Void. That means that you can store only one element there. Shallows provides a convenience .singleKey method to create it:

settings.retrieve { (result) in // ... } ">

let settings = DiskStorage.main.folder("settings", in: .applicationSupportDirectory)
    .mapJSONDictionary()
    .singleKey("settings") // Storage
settings.retrieve { (result) in
    // ...
}

Synchronous storage

Storages in Shallows are asynchronous by design. However, in some situations (for example, when scripting or testing) it could be useful to have synchronous storages. You can make any storage synchronous by calling .makeSyncStorage() on it:

let existing = try strings.retrieve(forKey: "hello") try strings.set(existing.uppercased(), forKey: "hello") ">

let strings = DiskStorage.main.folder("strings", in: .cachesDirectory)
    .mapString(withEncoding: .utf8)
    .makeSyncStorage() // SyncStorage
let existing = try strings.retrieve(forKey: "hello")
try strings.set(existing.uppercased(), forKey: "hello")

Mutating value for key

Shallows provides a convenient .update method on storages:

let arrays = MemoryStorage<String, [Int]>()
arrays.update(forKey: "some-key", { $0.append(10) })

Zipping storages

Zipping is a very powerful feature of Shallows. It allows you to compose your storages in a way that you get result only when both of them completes for your request. For example:

let strings = MemoryStorage<String, String>()
let numbers = MemoryStorage<String, Int>()
let zipped = zip(strings, numbers) // Storage
zipped.retrieve(forKey: "some-key") { (result) in
    if let (string, number) = try? result.get() {
        print(string)
        print(number)
    }
}
zipped.set(("shallows", 3), forKey: "another-key")

Isn't it nice?

Recovering from errors

You can protect your storage instance from failures using fallback(with:) or defaulting(to:) methods:

let storage = MemoryStorage<String, Int>()
let protected = storage.fallback(with: { error in
    switch error {
    case MemoryStorageError.noValue:
        return 15
    default:
        return -1
    }
})

let storage = MemoryStorage<String, Int>()
let defaulted = storage.defaulting(to: -1)

This is especially useful when using update method:

let storage = MemoryStorage<String, [Int]>()
storage.defaulting(to: []).update(forKey: "first", { $0.append(10) })

That means that in case of failure retrieving existing value, update will use default value of [] instead of just failing the whole update.

Using `NSCacheStorage`

NSCache is a tricky class: it supports only reference types, so you're forced to use, for example, NSData instead of Data and so on. To help you out, Shallows provides a set of convenience extensions for legacy Foundation types:

let nscache = NSCacheStorage<NSURL, NSData>()
    .toNonObjCKeys()
    .toNonObjCValues() // Storage

Making your own storage

To create your own caching layer, you should conform to StorageProtocol. That means that you should define these two methods:

func retrieve(forKey key: Key, completion: @escaping (Result<Value, Error>) -> ())
func set(_ value: Value, forKey key: Key, completion: @escaping (Result<Void, Error>) -> ())

Where Key and Value are associated types.

NOTE: Please be aware that you are responsible for the thread-safety of your implementation. Very often retrieve and set will not be called from the main thread, so you should make sure that no race conditions will occur.

To use it as Storage instance, simply call .asStorage() on it:

let storage = MyStorage().asStorage()

You can also conform to a ReadOnlyStorageProtocol only. That way, you only need to define a retrieve(forKey:completion:) method.

Installation

Swift Package Manager

Starting with Xcode 11, Shallows is officially available only via Swift Package Manager.

In Xcode 11 or greater, in you project, select: File > Swift Packages > Add Pacakage Dependency

In the search bar type

https://github.com/dreymonde/Shallows

Then proceed with installation.

If you can't find anything in the panel of the Swift Packages you probably haven't added yet your github account. You can do that under the Preferences panel of your Xcode, in the Accounts section.

For command-line based apps, you can just add this directly to your Package.swift file:

dependencies: [
    .package(url: "https://github.com/dreymonde/Shallows", from: "0.11.0"),
]

Manual

Of course, you always have an option of just copying-and-pasting the code.

Deprecated dependency managers

Last Shallows version to support Carthage and Cocoapods is 0.10.0. Carthage and Cocoapods will no longer be officially supported.

Carthage:

0.10.0 ">

github "dreymonde/Shallows" ~> 0.10.0

Cocoapods:

pod 'Shallows', '~> 0.10.0'

Comments

3-layer composition?
I'm trying to build a 3-layer, memory+disk+server (Firebase), composition, but I'm getting a "Type of expression is ambiguous without more context" error on the first line.

let myObjectMemoryCache = MemoryStorage<Filename, MyObject>() // <-- ambiguous type error .combined(with: DiskStorage.main.folder("myObjects", in: .documentDirectory).mapJSONObject(MyObject.self)) .combined(with: RemoteStorage.main.collection("myObjects").mapJSONObject(MyObject.self))

I'll admit, I'm really not sure how I'm supposed to do this. It would be great to have a little more info on how these kinds of custom compositions should be implemented. (I should mention that I do have a running memory+disk version that works fine, based on the examples you provide—I'm just not sure how to add another layer.)
opened by jbmaxwell 14
How to handle success in StorageProtocol set(value:forKey:completion:)

Stupid question, but I'm finally getting around to updating all my Shallows-related code to support the latest SwiftPM version, and I'm not sure how to handle a successful result in set(value:forKey:completion:). All I really want is to be able to fire some function in the caller on success, but I'm not clear on using ShallowsResult<Void> in this context... To clarify: my confusion is about implementing it in my custom Storage.

opened by jbmaxwell 13
Archiving fails in Swift5
I'm getting a very frustrating build error of Shallows / my adoption of Shallows that only occurs when archiving my application. I can run it just fine in debug mode with Swift 5 and it used to work fine to archive it when I was using Swift 4. But now I get a very weird build error that I don't understand. @dreymonde have you or anyone else using Shallows seen this problem or know what the issue is caused by? `CompileSwift normal armv7 ......... :0: error: fatal error encountered while reading from module 'Shallows'; please file a bug report with your project and the crash log

*** DESERIALIZATION FAILURE (please include this section in any bug report) *** (see "While..." info below) Stack dump: 0. Program arguments: ..........

While running pass #1 SILModuleTransform "PerformanceSILLinker".

While deserializing SIL function "$s8Shallows15StorageProtocolPAAE02asB0AA0B0Vy3KeyQz5ValueQzGyF" 0 swift 0x000000010705eee3 PrintStackTraceSignalHandler(void*) + 51 1 swift 0x000000010705e6bc SignalHandler(int) + 348 2 libsystem_platform.dylib 0x00007fff68619b5d _sigtramp + 29 3 libsystem_platform.dylib 000000000000000000 _sigtramp + 2543740096 4 libsystem_c.dylib 0x00007fff684d36a6 abort + 127 5 swift 0x00000001041de1e0 swift::ModuleFile::fatal(llvm::Error) + 1856 6 swift 0x0000000104204560 swift::SILDeserializer::readSILFunctionChecked(llvm::PointerEmbeddedInt<unsigned int, 31>, swift::SILFunction*, llvm::StringRef, bool, bool) + 9328 7 swift 0x00000001042158e1 swift::SILDeserializer::lookupSILFunction(swift::SILFunction*) + 673 8 swift 0x0000000103ecc69b swift::SILLinkerVisitor::maybeAddFunctionToWorklist(swift::SILFunction*) + 107 9 swift 0x0000000103ecc984 swift::SILLinkerVisitor::processFunction(swift::SILFunction*) + 292 10 swift 0x0000000103dabfe7 (anonymous namespace)::SILLinker::run() + 263 11 swift 0x0000000103cb63e8 swift::SILPassManager::execute() + 7416 12 swift 0x0000000102ec3668 performCompile(swift::CompilerInstance&, swift::CompilerInvocation&, llvm::ArrayRef<char const*>, int&, swift::FrontendObserver*, swift::UnifiedStatsReporter*) + 50280 13 swift 0x0000000102eb392e swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 6862 14 swift 0x0000000102e519ce main + 1246 15 libdyld.dylib 0x00007fff6842e3d5 start + 1 16 libdyld.dylib 0x0000000000000069 start + 2545753237`
opened by snoozemoose 13
Purge from memoryCache (while leaving in composed diskCache)?

Just curious whether there's a simple way (that I haven't seen) to remove an item from a memoryCache, while leaving it in the diskCache (I have a disk+memory composition). I want to be able to free up memory (on iOS), but retain the option to quickly reload from diskCache.

Thanks in advance.

opened by jbmaxwell 10
Memory limitations and responding to warnings with combined cache?

For the combined example of Memory cache with Disk Storage cache, I assume the memory cache wouldn't be cleared automatically if didReceiveMemoryWarning? If not, I can't see a way to manually clear the memory cache. Any suggestions? or have i missed something obvious, thanks.

opened by serendipityapps 5
Cache Expiration
Hi, very nice library. It's API has given me a lot of ideas about how to approach composition in my own app.

I've recently integrated with an ObjC caching library (SDWebImage). It supports cache expiration, and I was wondering if you planned on providing such functionality in this library?

It seems like some of this would be supported by making use of combined storages and possibly the "zipped" functionality. So you could have one storage contain keys and "inserted on" dates while another storage contains keys and the values you care about (for example, images). Cleaning up of expired keys could be handled on an as-needed basis by iterating through all the keys and comparing their "inserted on" dates with some expiry date.

Where I'm unsure, is how you'd implement something like a "Least Recently Used" strategy - wherein accessing a key updates it's "inserted on" date (you'd probably rename it to "updated on").

Imagine the following:

LRU Image Storage = Date Storage zipped with (In Memory Image Storage+Disk Image Storage)

If I read from this LRU Image Storage, I would want it to get from the In Memory storage first followed by the Disk storage if needed (that would be using the default pull strategy for combined storages). But how would I go about updating the entry in the Date Storage as the read occurred? I suppose a brute-force way to do so would be to just re-insert the image that was returned during the read, but that seems non-ideal from a usability/performance stand-point.
opened by nmccann 5
Cocoapods?

Very nice library you have here. I see the README says one installs through Carthage, but I also see a Shallows.podspec implying (possible eventual) Cocoapods support. Is that planned, or should the .podspec be removed?

Congrats on the iOS Dev Weekly inclusion too.

opened by scottcc 5
Memory cache returns nothing

I have a strange case where a Storage.retrieve completely skips its completion handler. The cache is composed of a MemoryStorage and a DiskStorage. I wish I could say more about it, but there's not much to say. I can see the Key go in, but the completion gets skipped entirely... ??

[EDIT: This wouldn't necessarily be a huge problem, except that I have a DispatchGroup.leave() waiting on the completion...]

opened by jbmaxwell 4
Clear object from MemoryStorage?

I've been debugging some memory usage issues and just noticed/remembered that Shallows is keeping my objects in a MemoryStorage, pretty much permanently. Is there an easy way to clear objects from MemoryStorage, or clear the MemoryStorage completely?

opened by jbmaxwell 3
Added static and dynamic library definitions

This change allows to link Shallows dynamically which makes it possible to use Shallows both in a framework as well as in the main app project simultaneously. In Xcode < 13.4 all we got was a warning at run time but in Xcode 13.4 the build process fails.

opened by snoozemoose 3
Base64 encoding isn't great for filenames
"/" is a valid base64 character and invalid for filenames. I recommend you use Base64URL encoding. This is probably just adding the following to lines:

.replacingOccurrences(of: "+", with: "-") .replacingOccurrences(of: "/", with: "_")
opened by GeorgeLyon 2
Clearer Error handling
Would it be possible to define a common Error enum for the StorageProtocol? Seems that now the errors are simply Error protocol, so making switch cases to determine the real cause is difficult.

For example

enum StorageError { case general(error: Error) case write(error: Error) case noValue(for: Key) }
opened by ollitapa 3

Releases(0.13.1)

0.13.1(Apr 23, 2022)

Source code(tar.gz)
Source code(zip)
0.11.0(Nov 15, 2019)

Starting with 0.11.0, Swift Package Manager is the only officially supported dependency manager for Shallows. Main reason — maintaining Carthage and especially Cocoapods is not trivial, which can sometimes lead to a Podfile not being up-to-date, for example. You can use Cocoapods/Carthage and Swift PM hand in hand, so I hope the transition will not be too painful

THIS UPDATE IS A BREAKING CHANGE Some APIs were slightly updated/removed. However, those are mostly non-primary APIs, and I expect that currently the very little amount of code depends on it. If you have problems migrating to 0.11.0, please, open an issue and I'll be glad to help!
Source code(tar.gz)
Source code(zip)
0.9.0(Feb 22, 2018)

This update aims to simplify the API by removing techniques that haven't proven themselves worthy. It also focuses on making the codebase more readable and easier to understand, and also refines some names and implementation details.

THIS UPDATE IS A BREAKING CHANGE Some API were deprecated, some - made unavailable. However, those are mostly non-primary APIs, and I expect that currently the very little amount of code depends on it. If you have problems migrating to 0.9.0, please, open an issue and I'll be glad to help!
Source code(tar.gz)
Source code(zip)
0.8.0(Jan 19, 2018)

This update drastically updates to Shallows way of disk storage. FileSystemStorage and family is now deprecated in favor of much more universal DiskStorage and DiskFolderStorage. FileSystemStorage will be removed from the codebase in future, but is still available for now. If you have trouble migrating to DiskFolderStorage — please, open an issue.
Source code(tar.gz)
Source code(zip)
0.6.0(Oct 11, 2017)

This release renames «cache» to «storage».
Source code(tar.gz)
Source code(zip)
0.5.0(Oct 11, 2017)

Source code(tar.gz)
Source code(zip)
0.3.0(Jun 7, 2017)
This release updates .combine API and adds some new minor features

Cache.combine now uses CacheCombinationPullStrategy and CacheCombinationSetStrategy instead of simple Bool flags, giving you more flexibility.

New method for CacheProtocol and ReadOnlyCache: fallback and defaulting

Minor improvements for RawFileSystemCache

.toNonObjCValues() for NSData and NSDate

Source code(tar.gz)
Source code(zip)
0.2.3(May 23, 2017)

This release adds the ability to specify DispatchQoS for FileSystemCache and RawFileSystemCache.
Source code(tar.gz)
Source code(zip)
0.2.2(May 9, 2017)
Renamed some APIs (for example, makeCache() -> asCache())

Introducing zip function for caches

Source code(tar.gz)
Source code(zip)

Owner

Oleg Dreyman

WWDC 2018 Scholarship Winner. iOS Engineer @nicephoton.

GitHub https://medium.com/anysuggestion

Typed key-value storage solution to store Codable types in various persistence layers with few lines of code!

?? Stores A typed key-value storage solution to store Codable types in various persistence layers like User Defaults, File System, Core Data, Keychain

94 Dec 31, 2022

A lightweight wrapper over UserDefaults/NSUserDefaults with an additional layer of AES-256 encryption

SecureDefaults for iOS, macOS Requirements • Usage • Installation • Contributing • Acknowledgments • Contributing • Author • License SecureDefaults is

216 Dec 22, 2022

Nora is a Firebase abstraction layer for FirebaseDatabase and FirebaseStorage

Nora is a Firebase abstraction layer for working with FirebaseDatabase and FirebaseStorage. Stop spending all that time cleaning up your view controll

273 Oct 15, 2022

Analytics layer abstraction, abstract analytics reporters and collect domain-driven analytic events.

?? Tentacles Current State: Work in Progress Documentation & Tests(100% completed, but needs refactoring and structuring) started but not done yet, im

3 Dec 2, 2022

SQLite.swift - A type-safe, Swift-language layer over SQLite3.

SQLite.swift provides compile-time confidence in SQL statement syntax and intent.

8.7k Jan 3, 2023

Disk is a powerful and simple file management library built with Apple's iOS Data Storage Guidelines in mind

3k Jan 3, 2023

Your Data Storage Troubleshooter 🛠

Your Data Storage Troubleshooter ?? Introduction StorageKit is a framework which reduces the complexity of managing a persistent layer. You can easily

231 Dec 29, 2022

Store and retrieve Codable objects to various persistence layers, in a couple lines of code!

tl;dr You love Swift's Codable protocol and use it everywhere, who doesn't! Here is an easy and very light way to store and retrieve Codable objects t

149 Dec 15, 2022

A type-safe, protocol-based, pure Swift database offering effortless persistence of any object

There are many libraries out there that aims to help developers easily create and use SQLite databases. Unfortunately developers still have to get bogged down in simple tasks such as writing table definitions and SQL queries. SwiftyDB automatically handles everything you don't want to spend your time doing.

489 Sep 9, 2022

A Generic CoreData Manager to accept any type of objects. Fastest way for adding a Database to your project.

QuickDB FileManager + CoreData ❗️ Save and Retrieve any thing in JUST ONE line of code ❗️ Fast usage dataBase to avoid struggling with dataBase comple

17 Sep 24, 2022

StorageManager - FileManager framework that handels Store, fetch, delete and update files in local storage

StorageManager - FileManager framework that handels Store, fetch, delete and update files in local storage. Requirements iOS 8.0+ / macOS 10.10+ / tvOS

47 Nov 3, 2022

An Objective-C wrapper for RocksDB - A Persistent Key-Value Store for Flash and RAM Storage.

ObjectiveRocks ObjectiveRocks is an Objective-C wrapper of Facebook's RocksDB - A Persistent Key-Value Store for Flash and RAM Storage. Current RocksD

56 Nov 5, 2022

An efficient, small mobile key-value storage framework developed by WeChat. Works on Android, iOS, macOS, Windows, and POSIX.

中文版本请参看这里 MMKV is an efficient, small, easy-to-use mobile key-value storage framework used in the WeChat application. It's currently available on Andr

15.4k Jan 6, 2023