Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a benchmark executable #79

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions Benchmarks/Benchmarks/ProtocolBenchmark/ProtocolBenchmark.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
//===----------------------------------------------------------------------===//
//
// This source file is part of the RediStack open source project
//
// Copyright (c) 2023 RediStack project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of RediStack project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//

@_spi(RESP3) import RediStack
import Benchmark
import NIOCore

let benchmarks = {
let resp3ArrayValueBuffer = ByteBuffer(string: "*2\r\n$3\r\nGET\r\n$7\r\nwelcome\r\n")
let resp3ArrayCount = 2

Benchmark("RESP3 Array Parsing") { benchmark in
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Benchmark("RESP3 Array Parsing") { benchmark in
Benchmark("RESP3 Array Parsing", configuration: .init(scalingFactor: .kilo)) { benchmark in
for _ in benchmark.scaledIterations {
try runRESP3ArrayParsing(
valueBuffer: resp3ArrayValueBuffer,
valueCount: resp3ArrayCount
)
}

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would suggest using such scaling factor (.kilo) for the RESP Array Parsing and RESP3 Array Parsing benchmarks, as they have very short runtimes. The scaling factor is used as an inner loop and the results will be scaled up accordingly for throughput if you update to a recent benchmark version.

try runRESP3ArrayParsing(
valueBuffer: resp3ArrayValueBuffer,
valueCount: resp3ArrayCount
)
}

let respArrayValueBuffer = ByteBuffer(string: "*2\r\n$3\r\nGET\r\n$7\r\nwelcome\r\n")
let respArrayCount = 2
Benchmark("RESP Array Parsing") { benchmark in
try runRESPArrayParsing(
valueBuffer: respArrayValueBuffer,
valueCount: respArrayCount
)
}

Benchmark("RESP3 Conversation") { benchmark in
try runRESP3Protocol()
}

Benchmark("RESP Conversation") { benchmark in
try runRESPProtocol()
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
//===----------------------------------------------------------------------===//
//
// This source file is part of the RediStack open source project
//
// Copyright (c) 2023 RediStack project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of RediStack project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//

import NIOCore
@_spi(RESP3) import RediStack

func runRESP3ArrayParsing(
valueBuffer: ByteBuffer,
valueCount: Int
) throws {
let token = RESP3Token.Unchecked(buffer: valueBuffer)

guard
case .array(let array) = try token.getValue(),
array.count == valueCount
else {
fatalError("\(#function) Test failed: Invalid test result")
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
//===----------------------------------------------------------------------===//
//
// This source file is part of the RediStack open source project
//
// Copyright (c) 2023 RediStack project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of RediStack project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//

import RediStack
import RESP3
import Foundation
import NIOCore
import NIOEmbedded

func runRESP3Protocol() throws {
let channel = EmbeddedChannel()

// Precalculate the server response
try channel.connect(to: .init(unixDomainSocketPath: "/fakeserver")).wait()
let serverReply = "Hello, world"
let redisReplyBuffer = ByteBuffer(string: "$\(serverReply.count)\r\n\(serverReply)\r\n")

// Client sends a command
// GET welcome
// TODO: Replace when we get RESP3 serialization
try channel.writeOutbound(ByteBuffer(string: "*2\r\n$3\r\nGET\r\n$7\r\nwelcome\r\n"))

// Server reads the command
_ = try channel.readOutbound(as: ByteBuffer.self)
// Server replies
try channel.writeInbound(redisReplyBuffer)

// Client reads the reply
guard var serverReplyBuffer = try channel.readInbound(as: ByteBuffer.self) else {
fatalError("Missing reply")
}

guard case .blobString(var blobString) = try RESP3Token(consuming: &serverReplyBuffer)?.value else {
fatalError("Invalid reply")
}

guard blobString.readString(length: blobString.readableBytes) == serverReply else {
fatalError("Invalid test result")
}

guard case .clean = try channel.finish() else {
fatalError("Test didn't exit cleanly")
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
//===----------------------------------------------------------------------===//
//
// This source file is part of the RediStack open source project
//
// Copyright (c) 2023 RediStack project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of RediStack project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//

import NIOCore
import RediStack

func runRESPArrayParsing(
valueBuffer: ByteBuffer,
valueCount: Int
) throws {
let translator = RESPTranslator()
var valueBuffer = valueBuffer
let value = try translator.parseBytes(from: &valueBuffer)
guard case .array(let result) = value, result.count == valueCount else {
fatalError("\(#function) Test failed: Invalid test result")
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
//===----------------------------------------------------------------------===//
//
// This source file is part of the RediStack open source project
//
// Copyright (c) 2023 RediStack project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of RediStack project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//

@_spi(RESP3) import RediStack
import Foundation
import NIOCore
import NIOEmbedded

func runRESPProtocol() throws {
let channel = EmbeddedChannel()
try channel.pipeline.addBaseRedisHandlers().wait()

// Precalculate the server response
try channel.connect(to: .init(unixDomainSocketPath: "/fakeserver")).wait()
var redisReplyBuffer = ByteBuffer()
let serverValue = "Hello, world"
let replyValue = RESPValue.simpleString(ByteBuffer(string: serverValue))
RESPTranslator().write(replyValue, into: &redisReplyBuffer)
let promise = channel.eventLoop.makePromise(of: RESPValue.self)

// Client sends a command
try channel.writeOutbound(RedisCommand(
message: .array([
.bulkString(ByteBuffer(string: "GET")),
.bulkString(ByteBuffer(string: "welcome")),
]),
responsePromise: promise
))

// Server reads the command
_ = try channel.readOutbound(as: ByteBuffer.self)
// Server replies
try channel.writeInbound(redisReplyBuffer)

// Client reads the reply
let serverReply = try promise.futureResult.wait()
guard serverReply.string == serverValue else {
fatalError("Invalid test result")
}

guard case .clean = try channel.finish() else {
fatalError("Test didn't exit cleanly")
}
}
40 changes: 40 additions & 0 deletions Benchmarks/Package.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
// swift-tools-version:5.9
//===----------------------------------------------------------------------===//
//
// This source file is part of the RediStack open source project
//
// Copyright (c) 2019-2023 RediStack project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of RediStack project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//

import PackageDescription

let package = Package(
name: "Benchmarks",
platforms: [
.macOS(.v13),
],
dependencies: [
.package(path: "../"),
.package(url: "https://github.com/ordo-one/package-benchmark.git", exact: "1.11.1"),
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest to use from:, or at least bump to a recent version (.e.g, 1.18.0) - 1.11.1 is quite old by now and NIO has from rather than exact.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realised this too, but had an issue with the latest build. I don't remember what it was, I believe my SPM caches were broken

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, @Joannis let me know if you have any issues with the package itself, 1.20 just released with quite a few fixes visavi 1.11.

],
targets: [
.executableTarget(
name: "ProtocolBenchmark",
dependencies: [
.product(name: "Benchmark", package: "package-benchmark"),
.product(name: "RediStack", package: "RediStack"),
],
path: "Benchmarks/ProtocolBenchmark",
plugins: [
.plugin(name: "BenchmarkPlugin", package: "package-benchmark")
]
),
]
)
5 changes: 3 additions & 2 deletions Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ let package = Package(
products: [
.library(name: "RediStack", targets: ["RediStack"]),
.library(name: "RediStackTestUtils", targets: ["RediStackTestUtils"]),
.library(name: "RedisTypes", targets: ["RedisTypes"])
.library(name: "RedisTypes", targets: ["RedisTypes"]),
],
dependencies: [
.package(url: "https://github.com/apple/swift-atomics.git", from: "1.1.0"),
Expand All @@ -40,7 +40,8 @@ let package = Package(
.product(name: "NIOSSL", package: "swift-nio-ssl"),
.product(name: "Atomics", package: "swift-atomics"),
.product(name: "Logging", package: "swift-log"),
.product(name: "Metrics", package: "swift-metrics")
.product(name: "Metrics", package: "swift-metrics"),
.target(name: "RESP3"),
]
),
.target(name: "RedisTypes", dependencies: ["RediStack"]),
Expand Down
Loading