How to deal with completely dynamic JSON responses - json

Question

Maybe someone in the community has had similar struggles and have come up with a workable solution.
We're currently working on a polyglot key/value store. Given this, we'll generally have no knowledge of what will be stored ahead of time.
Consider the following struct
struct Character : Codable, Equatable {
let name: String
let age: Int
let gender: Gender
let hobbies: [String]
static func ==(lhs: Character, rhs: Character) -> Bool {
return (lhs.name == rhs.name
&& lhs.age == rhs.age
&& lhs.gender == rhs.gender
&& lhs.hobbies == rhs.hobbies)
}
}
When sending/receiving Character entities over the wire, everything is fairly straight forward. The user can provide us the Type in which we can decode into.
However, we do have the ability to dynamically query the entities stored within the backend. For example, we can request the value of the 'name' property and have that returned.
This dynamism is a pain point. In addition to not knowing the type of the properties outside of the fact that they are Codable, the format that is returned can be dynamic as well.
Here's some examples of response for two different calls extracting properties:
{"value":"Bilbo"}
and
{"value":["[Ljava.lang.Object;",["Bilbo",111]]}
In some cases, it could be an equivalent of a dictionary.
Right now, I have the following structs for dealing with responses:
fileprivate struct ScalarValue<T: Decodable> : Decodable {
var value: T?
}
Using the Character example, the type passed to the decoder would be:
ScalarValue<Character>.self
However, for the single value, array, or dictionary case, I'm somewhat stuck.
I've started with something like:
fileprivate struct AnyDecodable: Decodable {
init(from decoder: Decoder) throws {
// ???
}
}
Based on the possible return types I've described above, I'm not sure if this is possible with the current API.
Thoughts?

Answer 1

Swift can definitely handle an arbitrary JSON decodable. This isn't the same thing as an arbitrary decodable. JSON can't encode all possible values. But this structure will decode anything that can be expressed in JSON, and from there you can explore it in a type-safe way without resorting to dangerous and awkward tools like Any.
enum JSON: Decodable, CustomStringConvertible {
var description: String {
switch self {
case .string(let string): return "\"\(string)\""
case .number(let double):
if let int = Int(exactly: double) {
return "\(int)"
} else {
return "\(double)"
}
case .object(let object):
return "\(object)"
case .array(let array):
return "\(array)"
case .bool(let bool):
return "\(bool)"
case .null:
return "null"
}
}
var isEmpty: Bool {
switch self {
case .string(let string): return string.isEmpty
case .object(let object): return object.isEmpty
case .array(let array): return array.isEmpty
case .null: return true
case .number, .bool: return false
}
}
struct Key: CodingKey, Hashable, CustomStringConvertible {
var description: String {
return stringValue
}
var hashValue: Int { return stringValue.hash }
static func ==(lhs: JSON.Key, rhs: JSON.Key) -> Bool {
return lhs.stringValue == rhs.stringValue
}
let stringValue: String
init(_ string: String) { self.stringValue = string }
init?(stringValue: String) { self.init(stringValue) }
var intValue: Int? { return nil }
init?(intValue: Int) { return nil }
}
case string(String)
case number(Double) // FIXME: Split Int and Double
case object([Key: JSON])
case array([JSON])
case bool(Bool)
case null
init(from decoder: Decoder) throws {
if let string = try? decoder.singleValueContainer().decode(String.self) { self = .string(string) }
else if let number = try? decoder.singleValueContainer().decode(Double.self) { self = .number(number) }
else if let object = try? decoder.container(keyedBy: Key.self) {
var result: [Key: JSON] = [:]
for key in object.allKeys {
result[key] = (try? object.decode(JSON.self, forKey: key)) ?? .null
}
self = .object(result)
}
else if var array = try? decoder.unkeyedContainer() {
var result: [JSON] = []
for _ in 0..<(array.count ?? 0) {
result.append(try array.decode(JSON.self))
}
self = .array(result)
}
else if let bool = try? decoder.singleValueContainer().decode(Bool.self) { self = .bool(bool) }
else {
self = .null
}
}
var objectValue: [String: JSON]? {
switch self {
case .object(let object):
let mapped: [String: JSON] = Dictionary(uniqueKeysWithValues:
object.map { (key, value) in (key.stringValue, value) })
return mapped
default: return nil
}
}
var arrayValue: [JSON]? {
switch self {
case .array(let array): return array
default: return nil
}
}
subscript(key: String) -> JSON? {
guard let jsonKey = Key(stringValue: key),
case .object(let object) = self,
let value = object[jsonKey]
else { return nil }
return value
}
var stringValue: String? {
switch self {
case .string(let string): return string
default: return nil
}
}
var doubleValue: Double? {
switch self {
case .number(let number): return number
default: return nil
}
}
var intValue: Int? {
switch self {
case .number(let number): return Int(number)
default: return nil
}
}
subscript(index: Int) -> JSON? {
switch self {
case .array(let array): return array[index]
default: return nil
}
}
var boolValue: Bool? {
switch self {
case .bool(let bool): return bool
default: return nil
}
}
}
With this, you can do things like:
let bilboJSON = """
{"value":"Bilbo"}
""".data(using: .utf8)!
let bilbo = try! JSONDecoder().decode(JSON.self, from: bilboJSON)
bilbo["value"] // "Bilbo"
let javaJSON = """
{"value":["[Ljava.lang.Object;",["Bilbo",111]]}
""".data(using: .utf8)!
let java = try! JSONDecoder().decode(JSON.self, from: javaJSON)
java["value"]?[1] // ["Bilbo", 111]
java["value"]?[1]?[0]?.stringValue // "Bilbo" (as a String rather than a JSON.string)
The proliferation of ? is somewhat ugly, but using throws on this doesn't really make the interface much nicer in my experiments (particularly because subscripts can't throw). Some tweaking may be advisable based on your particular use cases.

Answer 2

I wrote an AnyCodable struct myself for this purpose:
struct AnyCodable: Decodable {
var value: Any
struct CodingKeys: CodingKey {
var stringValue: String
var intValue: Int?
init?(intValue: Int) {
self.stringValue = "\(intValue)"
self.intValue = intValue
}
init?(stringValue: String) { self.stringValue = stringValue }
}
init(value: Any) {
self.value = value
}
init(from decoder: Decoder) throws {
if let container = try? decoder.container(keyedBy: CodingKeys.self) {
var result = [String: Any]()
try container.allKeys.forEach { (key) throws in
result[key.stringValue] = try container.decode(AnyCodable.self, forKey: key).value
}
value = result
} else if var container = try? decoder.unkeyedContainer() {
var result = [Any]()
while !container.isAtEnd {
result.append(try container.decode(AnyCodable.self).value)
}
value = result
} else if let container = try? decoder.singleValueContainer() {
if let intVal = try? container.decode(Int.self) {
value = intVal
} else if let doubleVal = try? container.decode(Double.self) {
value = doubleVal
} else if let boolVal = try? container.decode(Bool.self) {
value = boolVal
} else if let stringVal = try? container.decode(String.self) {
value = stringVal
} else {
throw DecodingError.dataCorruptedError(in: container, debugDescription: "the container contains nothing serialisable")
}
} else {
throw DecodingError.dataCorrupted(DecodingError.Context(codingPath: decoder.codingPath, debugDescription: "Could not serialise"))
}
}
}
extension AnyCodable: Encodable {
func encode(to encoder: Encoder) throws {
if let array = value as? [Any] {
var container = encoder.unkeyedContainer()
for value in array {
let decodable = AnyCodable(value: value)
try container.encode(decodable)
}
} else if let dictionary = value as? [String: Any] {
var container = encoder.container(keyedBy: CodingKeys.self)
for (key, value) in dictionary {
let codingKey = CodingKeys(stringValue: key)!
let decodable = AnyCodable(value: value)
try container.encode(decodable, forKey: codingKey)
}
} else {
var container = encoder.singleValueContainer()
if let intVal = value as? Int {
try container.encode(intVal)
} else if let doubleVal = value as? Double {
try container.encode(doubleVal)
} else if let boolVal = value as? Bool {
try container.encode(boolVal)
} else if let stringVal = value as? String {
try container.encode(stringVal)
} else {
throw EncodingError.invalidValue(value, EncodingError.Context.init(codingPath: [], debugDescription: "The value is not encodable"))
}
}
}
}
It works with nested dictionaries/arrays too. You can try it with any json in a playground.
let decoded = try! JSONDecoder().decode(AnyCodable.self, from: jsonData)

Answer 3

Yes, is possible to achieve what you described via the existing Codable API, and in an elegant manner I'd say (though I might be subjective here since I'm talking about my code :) ).
Let try to figure out what is needed for this task:
First things first, you need to declare all properties as optional. This is needed as the decoder is likely to have to deal with partial responses.
struct Character: Codable {
let name: String?
let age: Int?
let hobbies: [String]?
}
Next, we need a way to figure out how to map the struct properties to the various fields from the partial JSONs. Luckily the Codable API can help us here via the CodingKeys enum:
enum CodingKeys: String, CodingKey {
case name
case age
case hobbies
}
The first tricky part is to somehow convert the CodingKeys enum into an array of strings, that we can use for the array response - {"value":["[Ljava.lang.Object;",["Bilbo",111]]}. We are in luck here, there are various sources on the internet and SO that address the question of getting all cases of an enum. My preferred solutions is the RawRepresentable extension, since CodingKey is raw representable and it's raw value is a String:
// Adds support for retrieving all enum cases. Since we refer a protocol here,
// theoretically this method can be called on other types than enum
public extension RawRepresentable {
static var enumCases: [Self] {
var caseIndex: Int = 0
return Array(AnyIterator {
defer { caseIndex += 1 }
return withUnsafePointer(to: &caseIndex) {
$0.withMemoryRebound(to: Self.self, capacity: 1) { $0.pointee }
}
})
}
}
We're almost there, but we need some more work before we can decode.
Now that we have a Decodable type, a list of coding keys to use, we need a decoder that makes use of these. But before that, we need to be able to recognise types that can be partially decoded. Let's add a new protocol
protocol PartiallyDecodable: Decodable {
associatedtype PartialKeys: RawRepresentable
}
and make Character conform to it
struct Character : Codable, PartiallyDecodable {
typealias PartialKeys = CodingKeys
The finishing piece is the decoding part. We can reuse the JSONDecoder that comes with the standard library:
// Tells the form of data the server sent and we want to decode:
enum PartialDecodingStrategy {
case singleKey(String)
case arrayOfValues
case dictionary
}
extension JSONDecoder {
// Decodes an object by using a decoding strategy
func partialDecode<T>(_ type: T.Type, withStrategy strategy: PartialDecodingStrategy, from data: Data) throws -> T where T : PartiallyDecodable, T.PartialKeys.RawValue == String {
Connecting all of the above results in the following infrastructure:
// Adds support for retrieving all enum cases. Since we refer a protocol here,
// theoretically this method can be called on other types than enum
public extension RawRepresentable {
static var enumCases: [Self] {
var caseIndex: Int = 0
return Array(AnyIterator {
defer { caseIndex += 1 }
return withUnsafePointer(to: &caseIndex) {
$0.withMemoryRebound(to: Self.self, capacity: 1) { $0.pointee }
}
})
}
}
protocol PartiallyDecodable: Decodable {
associatedtype PartialKeys: RawRepresentable
}
// Tells the form of data the server sent and we want to decode:
enum PartialDecodingStrategy {
case singleKey(String)
case arrayOfValues
case dictionary
}
extension JSONDecoder {
// Decodes an object by using a decoding strategy
func partialDecode<T>(_ type: T.Type, withStrategy strategy: PartialDecodingStrategy, from data: Data) throws -> T where T : PartiallyDecodable, T.PartialKeys.RawValue == String {
guard let partialJSON = (try? JSONSerialization.jsonObject(with: data, options: [])) as? [AnyHashable:Any] else {
throw DecodingError.dataCorrupted(DecodingError.Context(codingPath: [], debugDescription: "Invalid JSON"))
}
guard let value = partialJSON["value"] else {
throw DecodingError.dataCorrupted(DecodingError.Context(codingPath: [], debugDescription: "Missing \"value\" key"))
}
let processedJSON: [AnyHashable:Any]
switch strategy {
case let .singleKey(key):
processedJSON = [key:value]
case .arrayOfValues:
guard let values = value as? [Any],
values.count == 2,
let properties = values[1] as? [Any] else {
throw DecodingError.dataCorrupted(DecodingError.Context(codingPath: [], debugDescription: "Invalid JSON: expected a 2 elements array for the \"value\" key"))
}
processedJSON = zip(T.PartialKeys.enumCases, properties)
.reduce(into: [:]) { $0[$1.0.rawValue] = $1.1 }
case .dictionary:
guard let dict = value as? [AnyHashable:Any] else {
throw DecodingError.dataCorrupted(DecodingError.Context(codingPath: [], debugDescription: "Invalid JSON: expected a dictionary for the \"value\" key"))
}
processedJSON = dict
}
return try decode(type, from: JSONSerialization.data(withJSONObject: processedJSON, options: []))
}
}
We want to be able to partially decode Character, so we make it adopt all the required protocols:
struct Character: Codable, PartiallyDecodable {
typealias PartialKeys = CodingKeys
let name: String?
let age: Int?
let hobbies: [String]?
enum CodingKeys: String, CodingKey {
case name
case age
case hobbies
}
}
Now the fun part, let's test it:
let decoder = JSONDecoder()
let jsonData1 = "{\"value\":\"Bilbo\"}".data(using: .utf8)!
print((try? decoder.partialDecode(Character.self,
withStrategy: .singleKey(Character.CodingKeys.name.rawValue),
from: jsonData1)) as Any)
let jsonData2 = "{\"value\":[\"[Ljava.lang.Object;\",[\"Bilbo\",111]]}".data(using: .utf8)!
print((try? decoder.partialDecode(Character.self,
withStrategy: .arrayOfValues,
from: jsonData2)) as Any)
let jsonData3 = "{\"value\":{\"name\":\"Bilbo\",\"age\":111,\"hobbies\":[\"rings\"]}}".data(using: .utf8)!
print((try? decoder.partialDecode(Character.self,
withStrategy: .dictionary,
from: jsonData3)) as Any)
As we might expect, the output is the following:
Optional(MyApp.Character(name: Optional("Bilbo"), age: nil, hobbies: nil))
Optional(MyApp.Character(name: Optional("Bilbo"), age: Optional(111), hobbies: nil))
Optional(MyApp.Character(name: Optional("Bilbo"), age: Optional(111), hobbies: Optional(["rings"])))
As we can see, with the proper infrastructure laid out, the only requirements for a type to be partially decodable is to conform to PartiallyDecodable and to have an enum that says which keys to decode. These requirements are easy to be followed.