diff --git a/.gitignore b/.gitignore index f4b02fada..bb4126eab 100644 --- a/.gitignore +++ b/.gitignore @@ -42,3 +42,5 @@ firebase.json # next-sitemap sitemap*.xml robots.txt + +public/images/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e657b2af7..a2d9c4b66 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -26,7 +26,7 @@ Storj's documentation is written using Markdown. See Github's [basic markdown gu The directory structure of the documentation follows Next.js [app directory routing](https://nextjs.org/docs/app/building-your-application/routing/defining-routes) conventions. This means that the URL path for a page corresponds to its location in the directory structure. -For example, a page located at `dcs/guide/page.md` would be accessible at `https://docs.storj.io/docs/guide`. +For example, a page located at `dcs/guide/page.md` would be accessible at `https://storj.dev/docs/guide`. Every documentation page must be named `page.md`. @@ -36,7 +36,7 @@ The front matter is a section at the beginning of each Markdown file (page.md) t **title:** The title of the article or page. This is typically displayed at the top of the page. -**docId:** A unique identifier for the document. This can be used for internal tracking and linking. See [Internal Linking](/CONTRIBUTING.md#internal-linking) +**docId:** A unique identifier for the document. This can be used for internal tracking and linking. See [Internal Linking](/CONTRIBUTING.md#internal-linking). If you're making a new page, generate a new unique ID with `pwgen -1 16` or something similar. **redirects:** A list of URLs that should redirect to this page. This is useful for maintaining links when a page's URL changes or for creating aliases for a page. diff --git a/README.md b/README.md index f2fb459d2..4c81bec67 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Storj Docs -Source code for +Source code for The docs are built using [Tailwind CSS](https://tailwindcss.com) and [Next.js](https://nextjs.org). diff --git a/app/(blog)/blog/a-tale-of-two-copies/07035c56befc8dc1.png b/app/(blog)/blog/a-tale-of-two-copies/07035c56befc8dc1.png new file mode 100644 index 000000000..6cb83ccd4 Binary files /dev/null and b/app/(blog)/blog/a-tale-of-two-copies/07035c56befc8dc1.png differ diff --git a/app/(blog)/blog/a-tale-of-two-copies/51fb41f548601419.png b/app/(blog)/blog/a-tale-of-two-copies/51fb41f548601419.png new file mode 100644 index 000000000..7989fad7b Binary files /dev/null and b/app/(blog)/blog/a-tale-of-two-copies/51fb41f548601419.png differ diff --git a/app/(blog)/blog/a-tale-of-two-copies/9b285294e5712e47.jpeg b/app/(blog)/blog/a-tale-of-two-copies/9b285294e5712e47.jpeg new file mode 100644 index 000000000..9a7575c2d Binary files /dev/null and b/app/(blog)/blog/a-tale-of-two-copies/9b285294e5712e47.jpeg differ diff --git a/app/(blog)/blog/a-tale-of-two-copies/a5d148aee871d9eb.png b/app/(blog)/blog/a-tale-of-two-copies/a5d148aee871d9eb.png new file mode 100644 index 000000000..0d3269dc0 Binary files /dev/null and b/app/(blog)/blog/a-tale-of-two-copies/a5d148aee871d9eb.png differ diff --git a/app/(blog)/blog/a-tale-of-two-copies/eb4b808e55daa547.png b/app/(blog)/blog/a-tale-of-two-copies/eb4b808e55daa547.png new file mode 100644 index 000000000..e85167d50 Binary files /dev/null and b/app/(blog)/blog/a-tale-of-two-copies/eb4b808e55daa547.png differ diff --git a/app/(blog)/blog/a-tale-of-two-copies/page.md b/app/(blog)/blog/a-tale-of-two-copies/page.md new file mode 100644 index 000000000..c695aa810 --- /dev/null +++ b/app/(blog)/blog/a-tale-of-two-copies/page.md @@ -0,0 +1,481 @@ +--- +author: + name: Jeff Wendling +date: '2021-08-10 00:00:00' +heroimage: ./a5d148aee871d9eb.png +layout: blog +metadata: + description: It was the best of times, it was the worst of times. That's when I + hit a performance mystery that sent me down a multi-day rabbit hole of adventure. + I was writing some code to take some entries, append them into a fixed size in-memory + buffer, and then flush that buffer to disk when it was full. + title: A Tale of Two Copies +title: A Tale of Two Copies + +--- + +It was the best of times, it was the worst of times. That's when I hit a performance mystery that sent me down a multi-day rabbit hole of adventure. I was writing some code to take some entries, append them into a fixed size in-memory buffer, and then flush that buffer to disk when it was full. The main bit of code looked a little something like this: + + +```go +type Buffer struct { + fh *os.File + n uint + buf [numEntries]Entry +} + +func (b *Buffer) Append(ent Entry) error { + if b.n < numEntries-1 { + b.buf[b.n] = ent + b.n++ + return nil + } + return b.appendSlow(ent) +} +``` + + +with the idea being that when there's space in the buffer, we just insert the entry and increment a counter, and when we're full, it falls back to the slower path that writes to disk. Easy, right? Easy... + +## The Benchmark + +I had a question about what size the entries should be. The minimum size I could pack them into was 28 bytes, but that's not a nice power of 2 for alignment and stuff, so I wanted to compare it to 32 bytes. Rather than just relying on my intuition, I decided to write a benchmark. The benchmark would Append a fixed number of entries per iteration (100,000) and the only thing changing would be if the entry size was 28 or 32 bytes. + +Even if I'm not relying on my intuition, I find it fun and useful to try to predict what will happen anyway. And so, I thought to myself: + + +> Everyone knows that I/O is usually dominating over small CPU potential inefficiencies. The 28 byte version writes less data and does less flushes to disk than the 32 byte version. Even if it's somehow slower filling the memory buffer, which I doubt, that will be more than made up for by the extra writes that happen. + +Maybe you thought something similar, or maybe something completely different. Or maybe you didn't sign up to do thinking right now and just want me to get on with it. And so, I ran the following benchmark: + + +```go +func BenchmarkBuffer(b *testing.B) { + fh := tempFile(b) + defer fh.Close() + + buf := &Buffer{fh: fh} + now := time.Now() + ent := Entry{} + + b.ResetTimer() + + for i := 0; i < b.N; i++ { + fh.Seek(0, io.SeekStart) + + for i := 0; i < 1e5; i++ { + _ = buf.Append(ent) + } + _ = buf.Flush() + } + + b.ReportMetric(float64(time.Since(now).Nanoseconds())/float64(b.N)/1e5, "ns/key") + b.ReportMetric(float64(buf.flushes)/float64(b.N), "flushes") +} + +``` + + +## Confusion + +And here are the results: + + +``` +BenchmarkBuffer/28 734286 ns/op 171.0 flushes 7.343 ns/key +BenchmarkBuffer/32 436220 ns/op 196.0 flushes 4.362 ns/key + +``` + + + +That's right, a nearly 2x difference in performance where the benchmark writing to disk MORE is FASTER! + +![](./51fb41f548601419.png) + +*Me too, Nick Young. Me, too.* + +And so began my journey. The following is my best effort in remembering the long, strange trip I took diagnosing what I thought was happening. Spoiler alert: I was wrong a lot, and for a long time. + +# The Journey + +## CPU Profiles + +CPU profiles have a huge power to weight ratio. To collect them from a Go benchmark, all you have to do is specify `-cpuprofile=` + on the command line and that's it. So of course this is the first thing I reached for. + +One thing to keep in mind, though, is that Go benchmarks by default will try to run for a fixed amount of time, and if one benchmark takes longer to do its job vs another, you get less iterations of it. Since I wanted to compare the results more directly, I made sure to also pass a fixed number of iterations to the command with `-benchtime=2000x` +. + +So let's take a look at these profiles. First, the 32 byte version: + + +``` + . . 24:func (b *Buffer) Append(ent Entry) error { + 30ms 30ms 25: if b.n < numEntries-1 { + 110ms 110ms 26: b.buf[b.n] = ent + 90ms 90ms 27: b.n++ + . . 28: return nil + . . 29: } + 10ms 520ms 30: return b.appendSlow(ent) + . . 31:} + +``` + + +The first column shows the amount of time spent on that line just in the context of the shown function, and the second column is the amount of time spent on that line including any functions it may have called. + +From that, we can see that, as expected, most of the time is spent flushing to disk in appendSlow compared to writing to the in memory buffer. + +And now here's the 28 byte version: + + +``` + . . 24:func (b *Buffer) Append(ent Entry) error { + 20ms 20ms 25: if b.n < numEntries-1 { + 840ms 840ms 26: b.buf[b.n] = ent + 20ms 20ms 27: b.n++ + . . 28: return nil + . . 29: } + . 470ms 30: return b.appendSlow(ent) + . . 31:} + +``` + + + +A couple of things stand out to me here. First of all, WHAT? Second of all, it spends less time flushing to disk compared to the 32 byte version. That's at least expected because it does that less often (171 vs 196 times). And finally, WHAT? + +Maybe the penalty for writing unaligned memory was worse than I thought. Let's take a look at the assembly to see what instruction it's stuck on. + +## The Assembly + +Here's the section of code responsible for the 840ms on line 26 in the above profile: + + +``` + . . 515129: IMULQ $0x1c, CX, CX (1) + 90ms 90ms 51512d: LEAQ 0xc0(SP)(CX*1), CX (2) + . . 515135: MOVUPS 0x7c(SP), X0 (3) + 670ms 670ms 51513a: MOVUPS X0, 0(CX) (4) + 80ms 80ms 51513d: MOVUPS 0x88(SP), X0 (5) + . . 515145: MOVUPS X0, 0xc(CX) (6) + +``` + + +If you've never read assembly before, this may be a bit daunting, so I've numbered the lines and will provide a brief explanation. The most important bits to know are that `CX` +, `SP` + and `X0` + are registers, and the syntax `0x18(CX)` + means the value at address `CX + 0x18` +. Armed with that knowledge, we can understand the lines: + +1. Multiply the `CX` + register by `0x1c` + and store it into `CX` +. `0x1c` + is the hex encoding of the decimal value 28. +2. This computes the address we'll be storing the entry into. It computes `0xc0 + SP + (CX*1)` + and stores it into `CX` +. From this, we deduce that the start of the entry array is at `0xc0(SP)` +. +3. This loads 16 bytes starting at `0x7c(SP)` + and stores it into `X0` +. +4. This stores the 16 bytes we just loaded into `0(CX)` +. +5. This loads 16 bytes starting at `0x88(SP)` + and stores it into `X0` +. +6. This stores the 16 bytes we just loaded into `0xc(CX)` +. + +I don't know about you, but I saw no reason why line 4 should have so much weight compared to the other lines. So, I compared it to the 32 byte version to see if the generated code was different: + + +``` + 40ms 40ms 515129: SHLQ $0x5, CX + 10ms 10ms 51512d: LEAQ 0xc8(SP)(CX*1), CX + . . 515135: MOVUPS 0x80(SP), X0 + 10ms 10ms 51513d: MOVUPS X0, 0(CX) + 40ms 40ms 515140: MOVUPS 0x90(SP), X0 + 10ms 10ms 515148: MOVUPS X0, 0x10(CX) + +``` + + + +It looks like the only difference, aside from almost no time at all being spent in these instructions, is the SHLQ vs the IMULQ. The former is doing a "left shift" of 5, which effectivly multiplies by 2 to the 5th power, or 32, and the latter, as we previously saw, multiplies by 28. Could this possibly be the performance difference? + +## Pipelines and Ports + +Modern CPUs are complex beasts. Maybe you have the mental model that your CPU reads instructions in and executes them one at a time as I once did. That couldn't be further from the truth. Instead, they execute multiple instructions at once, possibly out of order, in a [pipeline](https://en.wikipedia.org/wiki/Instruction_pipelining). But it gets even better: they have limits on how many of each kind of instruction can be run simultaneously. This is done by the CPU having multiple "ports", and certain instructions require and can run on different subsets of these ports. + +So what does that have to do with IMULQ vs SHLQ? Well, you may have noticed that the LEAQ following the IMULQ/SHLQ has a multiply in it (`CX*1` +). But, because there aren't infinite ports, there must be a limited number of ports able to do multiplies. + +The LLVM project has lots of tools to help you understand what computers do, and one of them is a tool called [`llvm-mca`](https://www.llvm.org/docs/CommandGuide/llvm-mca.html#how-llvm-mca-works). Indeed, if we run the two first instructions of the 32 and 28 byte versions through `llvm-mca` +, it gives us an idea of what ports will be used when they are executed: + + +``` +Resource pressure by instruction (32 byte version): +[2] [3] [7] [8] Instructions: +0.50 - - 0.50 shlq $5, %rcx + - 0.50 0.50 - leaq 200(%rsp,%rcx), %rcx + +Resource pressure by instruction (28 byte version): +[2] [3] [7] [8] Instructions: + - 1.00 - - imulq $28, %rcx, %rcx + - - 1.00 - leaq 192(%rsp,%rcx), %rcx + +``` + + + +The numbers are what percent of the time each instruction ran on the port (here, numbered 2, 3, 7 and 8) when executed in a loop. + +So that's saying that in the 32 byte version, the SHLQ ran on port 2 half the time and port 8 the other half, and the LEAQ ran on port 3 half the time and port 7 the other half. This is implying that it can have 2 parallel executions at once. For example, on one iteration, it can use ports 2 and 3, and on the next iteration it can use ports 7 and 8, even if ports 2 and 3 are still being used. However, for the 28 byte version, the IMULQ must happen solely on port 3 due to the way the processor is built, which in turn limits the maximum throughput. + +And for a while, this is what I thought was happening. In fact, a first draft of this very blog post had that as the conclusion, but the more I thought about it, the less good of an explanation it seemed. + +![](./eb4b808e55daa547.png) + +*My first draft attempt* + +## Trouble in Paradise + +Here are some thoughts that you may be having: + +1. In the worst case, that can only be a 2x speed difference. +2. Aren't there other instructions in the loop? That has to make it so that it's much less than 2x in practice. +3. The 32 byte version spends 230ms in the memory section and the 28 byte version spends 880ms. +4. That is much bigger than 2x bigger. +5. Oh no. + +Well, maybe that last one was just me. With those doubts firmly in my mind, I tried to figure out how I could test to see if it was because of the IMULQ and SHLQ. Enter `perf` +. + +## Perf + +[`perf`](https://perf.wiki.kernel.org/index.php/Main_Page) is a tool that runs on linux that allows you to execute programs and expose some detailed counters that CPUs keep about how they executed instructions (and more!). Now, I had no idea if there was a counter that would let me see something like "the pipeline stalled because insufficient ports or whatever", but I did know that it had counters for like, everything. + +If this were a movie, this would be the part where the main character is shown trudging through a barren desert, sun blazing, heat rising from the earth, with no end in sight. They'd see a mirage oasis and jump in, gulping down water, and suddenly realize it was sand. + +A quick estimate shows that perf knows how to read over 700 different counters on my machine, and I feel like I looked at most of them. Take a look at [this huge table](https://perfmon-events.intel.com/skylake.html) if you're interested. I couldn't find any counters that could seem to explain the large difference in speed, and I was starting to get desparate. + +![](./9b285294e5712e47.jpeg) + +*A picture of me wading through all of the perf counters* + +## Binary Editing for Fun and Profit + +At this point, I had no idea what the problem was, but it sure seemed like it wasn't port contention like I thought. One of the only other things that I thought it could be was alignment. CPUs tend to like to have memory accessed at nice multiples of powers of 2, and 28 is not one of those, and so I wanted to change the benchmark to write 28 byte entries but at 32 byte offsets. + +Unfortunately, this wasn't as easy as I hoped. The code under test is very delicately balanced with respect to the Go compiler's inliner. Basically any changes to Append cause it to go over the threshold and stop it from being inlined, which really changes what's being executed. + +Enter binary patching. It turns out that in our case, the IMULQ instruction encodes to the same number of bytes as the SHLQ. Indeed, the IMULQ encodes as `486bc91c` +, and the SLHQ as `48c1e105` +. So it's just a simple matter of replacing those bytes and running the benchmark. I'll (for once) spare you the details of how I edited it (Ok, I lied: I hackily used `dd` +). The results sure did surprise me: + + +``` +BenchmarkBuffer/28@32 813529 ns/op 171.0 flushes 8.135 ns/key + +``` + + + +I saw the results and felt defeated. It wasn't the IMULQ making the benchmark go slow. That benchmark has no IMULQ in it. It wasn't due to unaligned writes. The slowest instruction was written with the same alignment as in the 32 byte version as we can see from the profiled assembly: + + +``` + . . 515129: SHLQ $0x5, CX + 60ms 60ms 51512d: LEAQ 0xc0(SP)(CX*1), CX + . . 515135: MOVUPS 0x7c(SP), X0 + 850ms 850ms 51513a: MOVUPS X0, 0(CX) + 120ms 120ms 51513d: MOVUPS 0x88(SP), X0 + . . 515145: MOVUPS X0, 0xc(CX) + +``` + + +What was left to try? + +# A Small Change + +Sometimes when I have no idea why something is slow, I try writing the same code but in a different way. That may tickle the compiler just right to cause it to change which optimizations it can or can't apply, giving some clues as to what's going on. So in that spirit I changed the benchmark to this: + + +```go +func BenchmarkBuffer(b *testing.B) { + // ... setup code + + for i := 0; i < b.N; i++ { + fh.Seek(0, io.SeekStart) + + for i := 0; i < 1e5; i++ { + _ = buf.Append(Entry{}) + } + _ = buf.Flush() + } + + // .. teardown code +} + +``` + + + +It's hard to spot the difference, but it changed to passing a new entry value every time instead of passing the `ent` + variable manually hoisted out of the loop. I ran the benchmarks again. + + +``` +BenchmarkBuffer/28 407500 ns/op 171.0 flushes 4.075 ns/key +BenchmarkBuffer/32 446158 ns/op 196.0 flushes 4.462 ns/key + +``` + + + +IT DID SOMETHING? How could that change possibly cause that performance difference? It's finally running faster than the 32 byte version! As usual, time to look at the assembly. + + +``` + 50ms 50ms 515109: IMULQ $0x1c, CX, CX + . . 51510d: LEAQ 0xa8(SP)(CX*1), CX + . . 515115: MOVUPS X0, 0(CX) + 130ms 130ms 515118: MOVUPS X0, 0xc(CX) + +``` + + + +It's no longer loading the value from the stack to store it into the array, and instead just storing directly into the array from the already zeroed register. But we know from all the pipeline analysis done earlier that the extra loads should effectively be free, and the 32 byte version confirms that. It didn't get any faster even though it also is no longer loading from the stack. + +So what's going on? + +## Overlapping Writes + +In order to explain this idea, it's important to show the assembly of the full inner loop instead of just the code that writes the entry to the in-memory buffer. Here's a cleaned up and annotated version of the slow 28 byte benchmark inner loop: + + +``` +loop: + INCQ AX (1) + CMPQ $0x186a0, AX + JGE exit + + MOVUPS 0x60(SP), X0 (2) + MOVUPS X0, 0x7c(SP) + MOVUPS 0x6c(SP), X0 + MOVUPS X0, 0x88(SP) + + MOVQ 0xb8(SP), CX (3) + CMPQ $0x248, CX + JAE slow + + IMULQ $0x1c, CX, CX (4) + LEAQ 0xc0(SP)(CX*1), CX + MOVUPS 0x7c(SP), X0 (5) + MOVUPS X0, 0(CX) + MOVUPS 0x88(SP), X0 + MOVUPS X0, 0xc(CX) + + INCQ 0xb8(SP) (6) + JMP loop + +slow: + // ... slow path goes here ... + +exit: + +``` + + +1. Increment `AX` + and compare it to 100,000 exiting if it's larger. +2. Copy 28 bytes on the stack from offsets `[0x60, 0x7c]` + to offsets `[0x7c, 0x98]` +. +3. Load the memory counter and see if we have room in the memory buffer +4. Compute where the entry will be written to in the in-memory buffer. +5. Copy 28 bytes on the stack at offsets `[0x7c, 0x98]` + into the in-memory buffer. +6. Increment the memory counter and loop again. + +Steps 4 and 5 are what we've been looking at up to now. + +If step 2 seems silly and redundant, that's because it is. There's no reason to copy a value on the stack to another location on the stack and then load from that copy on the stack into the in-memory buffer. Step 5 could have just used offsets `[0x60, 0x7c]` + instead and step 2 could have been eliminated. The Go compiler could be doing a better job here. + +But that shouldn't be why it's slow, right? The 32 byte code does almost the exact same silly thing and it goes fast, because of pipelines or pixie dust or something. What gives? + +There's one crucial difference: the writes in the 28 byte case overlap. The MOVUPS instruction writes 16 bytes at a time, and as everyone knows, 16 + 16 is usually more than 28. So step 2 writes to bytes `[0x7c, 0x8c]` + and then writes to bytes `[0x88, 0x98]` +. This means the range `[0x88, 0x8c]` + was written to twice. Here's a helpful ASCII diagram: + + +``` +0x7c 0x8c +├────────────────┤ +│ Write 1 (16b) │ +└───────────┬────┴──────────┐ + │ Write 2 (16b) │ + ├───────────────┤ + 0x88 0x98 + +``` + + +## Store Forwarding + +Remember how CPUs are complex beasts? Well it gets even better. An optimization that some CPUs do is they have something called a ["write buffer"](https://en.wikipedia.org/wiki/Write_buffer). You see, memory access is often the slowest part of what CPUs do. Instead of, you know, actually writing the memory when the instruction executes, CPUs place the writes into a buffer first. I think the idea is to coalesce a bunch of small writes into larger sizes before flushing out to the slower memory subsystem. Sound familiar? + +So now it has this write buffer buffering all of the writes. What happens if a read comes in for one of those writes? It would slow everything down if had to wait for that write to actually happen before reading it back out, so instead it tries to service the read from the write buffer directly if possible, and no one is the wiser. You clever little CPU. This optimization is called [store forwarding](https://easyperf.net/blog/2018/03/09/Store-forwarding). + +![](./07035c56befc8dc1.png) + +*My CPU buffering and reorganizing all of the writes* + +But what if those writes overlap? It turns out that, on my CPU at least, this inhibits that "store forwarding" optimization. There's even a perf counter that keeps track of when this happens: [ld\_blocks.store\_forward](https://perfmon-events.intel.com/index.html?pltfrm=skylake.html&evnt=LD_BLOCKS.STORE_FORWARD). + +Indeed, the documentation about that counter says + + +> Counts the number of times where store forwarding was prevented for a load operation. The most common case is a load blocked due to the address of memory access (partially) overlapping with a preceding uncompleted store. + +Here's how often that counter hits for the different benchmarks so far where "Slow" means that the entry is constructed outside of the loop, and "Fast" means that the entry is constructed inside of the loop on every iteration: + + +``` +BenchmarkBuffer/28-Slow 7.292 ns/key 1,006,025,599 ld_blocks.store_forward +BenchmarkBuffer/32-Slow 4.394 ns/key 1,973,930 ld_blocks.store_forward +BenchmarkBuffer/28-Fast 4.078 ns/key 4,433,624 ld_blocks.store_forward +BenchmarkBuffer/32-Fast 4.369 ns/key 1,974,915 ld_blocks.store_forward + +``` + + +Well, a billion is usually bigger than a million. Break out the champagne. + +# Conclusion + +After all of that, I have a couple of thoughts. + +Benchmarking is hard. People often say this, but maybe the only thing harder than benchmarking is adequately conveying how hard benchmarking is. Like, this was closer to the micro-benchmark than macro-benchmark side of things but still included performing millions of operations including disk flushes and actually measured a real effect. But at the same time, this would almost never be a problem in practice. It required the compiler to spill a constant value to the stack unnecessarily very closely to the subsequent read in a tight inner loop to notice. Doing any amount of real work to create the entries would cause this effect to vanish. + +A recurring theme as I learn more about how CPUs work is that the closer you get to the "core" of what it does, the leakier and more full of edge cases and hazards it becomes. Store forwarding not working if there was a partially overlapping write is one example. Another is that the caches aren't [fully associative](https://en.wikipedia.org/wiki/CPU_cache#Associativity), so you can only have so many things cached based on their memory address. Like, even if you have 1000 slots available, if all your memory accesses are multiples of some factor, they may not be able to use those slots. [This blog post](https://danluu.com/3c-conflict/) has a great discussion. Totally speculating, but maybe this is because you have less "room" to solve those edge cases when under ever tighter physical constraints. + +Before now, I've never been able to concretely observe the CPU slowing down from port exhaustion issues in an actual non-contrived setting. I still haven't. I've heard the adage that you can imagine every CPU instruction taking 0 cycles except for the ones that touch memory. As a first approximation, it seems pretty true. + +I've put up the full code sample in [a gist](https://gist.github.com/zeebo/4c9e28ac277c74ae450ad1bff8068f93) for your viewing/downloading/running/inspecting pleasure. + +Often, things are more about the journey than the destination, and I think that's true here, too. If you made it this far, thanks for coming along on the journey with me, and I hope you enjoyed it. Until next time. + + + diff --git a/app/(blog)/blog/automatically-store-your-tesla-sentry-mode-and-dashcam-videos-on-the-decentralized-cloud/94ca9aa0a874a87b.png b/app/(blog)/blog/automatically-store-your-tesla-sentry-mode-and-dashcam-videos-on-the-decentralized-cloud/94ca9aa0a874a87b.png new file mode 100644 index 000000000..3d8555684 Binary files /dev/null and b/app/(blog)/blog/automatically-store-your-tesla-sentry-mode-and-dashcam-videos-on-the-decentralized-cloud/94ca9aa0a874a87b.png differ diff --git a/app/(blog)/blog/automatically-store-your-tesla-sentry-mode-and-dashcam-videos-on-the-decentralized-cloud/page.md b/app/(blog)/blog/automatically-store-your-tesla-sentry-mode-and-dashcam-videos-on-the-decentralized-cloud/page.md new file mode 100644 index 000000000..77eaa3deb --- /dev/null +++ b/app/(blog)/blog/automatically-store-your-tesla-sentry-mode-and-dashcam-videos-on-the-decentralized-cloud/page.md @@ -0,0 +1,100 @@ +--- +author: + name: Krista Spriggs +date: '2021-06-15 00:00:00' +heroimage: ./94ca9aa0a874a87b.png +layout: blog +metadata: + description: You can automatically transfer Sentry Mode and Dashcam video clips + over WiFi to cloud storage and make room for more videos the next day. We used + a Raspberry Pi (a small, low cost, low power computer about the size of an Altoids + tin) plugged into the USB port in the dashboard to store the video files. When + the Tesla pulls into the garage at night, the Raspberry Pi connects via WiFi and + uploads all the videos to Storj DCS cloud storage, then clears off the drive for + use the next day. This will also work for videos recorded in Track Mode if you + have one of the performance models, making it easy to share any of the videos + with your friends. + title: Automatically Store Your Tesla Sentry Mode and Dashcam Videos on the Decentralized + Cloud +title: Automatically Store Your Tesla Sentry Mode and Dashcam Videos on the Decentralized + Cloud + +--- + +I have a 2019 Tesla M3 and I love the built in features to capture Dashcam and Sentry Mode footage for review later. The Sentry Mode feature captures 10 minutes of video when someone or something approaches your parked vehicle. Dashcam captures video when you're driving. It continuously stores the most recent hour of video and will also save the most recent 10 minutes of video when you press the Dashcam button or when you honk your horn. Where do those videos get saved? As it turns out, they get saved to a flash drive, if you have one plugged into one of the USB ports in the front console. + + +When you get home, you have to pull the flash drive out of the car and copy the video files to your computer to watch them or store them long term. It’s a fairly low-tech way to manage your data. If you don’t free up space on the flash drive, Sentry Mode will eventually fill up the storage device and your Dashcam feature will stop saving video. What if there was an easy way to save all your Sentry Mode and Dashcam videos automatically when you pull into your garage, and ensure you always have space for new videos? As it turns out, [this is a solved problem](https://github.com/marcone/teslausb)\* if you just want to copy the videos to a computer on your home network. We’ve taken this open source project maintained by GitHub user marcone and created by Reddit user drfrank, and connected it to store the videos on Storj DCS, a decentralized cloud storage service that is secure, private, and extremely affordable.  + +### TL;DR + +You can automatically transfer Sentry Mode and Dashcam video clips over WiFi to cloud storage and make room for more videos the next day. We used a Raspberry Pi (a small, low cost, low power computer about the size of an Altoids tin) plugged into the USB port in the dashboard to store the video files. When the Tesla pulls into the garage at night, the Raspberry Pi connects via WiFi and uploads all the videos to Storj DCS cloud storage, then clears off the drive for use the next day. This will also work for videos recorded in Track Mode if you have one of the performance models, making it easy to share any of the videos with your friends. + + +### So, how hard is it? + +Ok, so most Tesla owners tend to be pretty technical, so, if that describes you, this is a piece of cake, sorry, Pi. Here’s what you’ll need: + +* [Raspberry Pi Zero W : ID 3400 : $10.00](https://www.adafruit.com/product/3400) - we used a different model, but this is better. +* [Adafruit Raspberry Pi Zero Case : ID 3252 : $4.75](https://www.adafruit.com/product/3252) - it should look good - you can 3d print your own for extra credit. +* Y [Video microSDXC Card](https://www.amazon.com/SanDisk-Endurance-microSDXC-Adapter-Monitoring/dp/B07P4HBRMV) $37 - it’s very important to have high quality storage with high write endurance. This gives you room for a few days in case you don’t connect to WiFi and won't wear out too quickly. +* [USB A to Micro-B - 3 foot long](https://www.adafruit.com/product/592) - A USB Cable to plug into the car. +* [Storj DCS cloud storage](https://www.storj.io/) - Storj provides 25 GB for free and it’s only $0.004 per GB after that! Secure, private, and decentralized. +* Optional Items for easier setup - A [Mini HDMI to HDMI Cable - 5 feet : ID 2775 : $5.95](https://www.adafruit.com/product/2775) will make it easier to set everything up by connecting the Pi to a monitor. + +All in, you’re looking at right around $60 to get going. I’ll wait while you get everything ordered... + + +### Okay, what’s next? + +Assuming you have everything you need, we’ve published the [detailed, step-by-step instructions in this tutorial](docId:XjYoGwaE6ncc3xTICXOOu). In general, there are four main steps which we share in a brief overview below: + + +* Create your Storj DCS account +* Set up your Raspberry Pi +* Enable Sentry Mode and Dashcam Mode +* Drive around and honk your horn + +After that, you can just go home and park your car. Your Raspberry Pi will connect to your home WiFi and do the rest. You’ll be able to view and share your videos from Storj DCS. + +### Create your Storj DCS account + +Storj DCS is a decentralized cloud object storage service. Storj DCS is like the Airbnb for hard drives—thousands of people with extra hard drive space share that space with Storj, then Storj makes that space available to use for storing things like Tesla videos. Every file stored on Storj DCS is encrypted, encoded for redundancy, and divided into at least 80 pieces— each stored on a different hard drive all over the world, provided by people who share storage space and bandwidth in exchange for STORJ tokens. Of those 80 pieces, you only need any 29 to download your file. That’s a bit of an oversimplification, and it’s even better than it sounds, but you get the idea; cloud storage for your videos. We just released some new features and pretty amazing pricing on April 20 and now you can try it. + + +First, go to Storj.io and create an account. You get 25 GB of storage and bandwidth a month for free with your new account. After that, you’ll need to create an Access Grant and generate the S3-compatible gateway credentials for use in the application that uploads your data to Storj DCS. Follow the steps in the tutorial and save your gateway credentials for the next step. + + +### Set up your Raspberry Pi + +A Raspberry Pi is a mini computer. Setting one up is relatively straightforward. Once you assemble the parts, you can connect to it remotely or plug it in to a monitor keyboard and mouse. To get it up and running, you download an OS like Raspian, flash it to the SD card and boot it up. From there, it’s a matter of installing a few components, including Rclone, which is the application that will upload your videos to Storj DCS. Configure Rclone with the gateway credentials you created on your Storj DCS account. Once you have the Raspberry Pi working, shut it down, unplug everything and head out to your Tesla with the Raspberry Pi and the USB cable. + + +### Enable Sentry Mode and Dashcam Mode + +Once you’re in your car, plug the Raspberry Pi into one of the USB ports in the front console. (The ones in the back don’t work for this project.) The Raspberry Pi will store video files through and is also powered by the USB port, so it needs to stay plugged in. Now, you’ll need to enable Sentry Mode and Dashcam Mode. These features are not enabled by default. Follow the steps in the tutorial to enable those two features on your Tesla. Once the Raspberry Pi is plugged in and the features are enabled, you’re ready to see it in action. + + +### Drive around and honk your horn + +The easiest way to capture some video clips is to drive around and honk your horn. Of course, if you worked on this until late into the night, your neighbors may or may not be as excited to test it as you are, so honk responsibly. As an alternative, drive around and click the Dashcam button to save a clip. Really, it’s up to you, but just get some video footage. All of the videos generated by Sentry Mode and Dashcam Mode will be saved to the SD card in the Raspberry Pi. + + +Once you’ve got some video, it’s time for the real magic—go home. When you pull into your garage and your Raspberry Pi connects to your home WiFi, it will upload the trip’s videos to Storj DCS.  + + +As you drive around, honk your horn, capture Dashcam videos and accumulate Sentry Mode video. Upon return, the videos will be uploaded to your Storj DCS account. Every one of those videos will be encrypted, erasure coded and stored in pieces distributed across our network of 13,000 Storage Nodes (and growing). You can view, download or share those videos with your friends. We’ve shared a sample video from a Tesla belonging to a Storj team member. When you share a file though Storj DCS,  the link lets you see all the Storage Nodes storing pieces of your file and stream the file directly from the network. The tutorial also has the steps to share a file (hint: click the share button to create a secure and private link to share). + + +### That’s all there is to it + +If you’ve followed along and followed the steps in your tutorial, your Tesla will store your Sentry and Dashcam videos in the decentralized cloud for as long as you want or need them.  + + +Overall this was a really fun project to put together, and shows off yet another way that you can integrate with Storj DCS easily and quickly! + + + +*\* The code used in this tutorial is open source and uses, among other things,* [*RClone*](https://github.com/rclone/rclone) *which includes native support for Storj DCS. The GitHub Repository for the code is available at:* [*https://github.com/marcone/teslausb*](https://github.com/marcone/teslausb) *and the project was originally described on the* [*/r/teslamotors*](https://www.reddit.com/r/teslamotors/comments/9m9gyk/build_a_smart_usb_drive_for_your_tesla_dash_cam/) *subreddit.* + + diff --git a/app/(blog)/blog/choosing-cockroach-db-for-horizontal-scalability/hero.png b/app/(blog)/blog/choosing-cockroach-db-for-horizontal-scalability/hero.png new file mode 100644 index 000000000..e2f8e326c Binary files /dev/null and b/app/(blog)/blog/choosing-cockroach-db-for-horizontal-scalability/hero.png differ diff --git a/app/(blog)/blog/choosing-cockroach-db-for-horizontal-scalability/page.md b/app/(blog)/blog/choosing-cockroach-db-for-horizontal-scalability/page.md new file mode 100644 index 000000000..556382791 --- /dev/null +++ b/app/(blog)/blog/choosing-cockroach-db-for-horizontal-scalability/page.md @@ -0,0 +1,175 @@ +--- +author: + name: Krista Spriggs and Jessica Grebenschikov +date: '2020-08-11 00:00:00' +heroimage: ./hero.png +layout: blog +metadata: + description: Here at Storj Labs we just migrated our production databases from + PostgreSQL to CockroachDB. We want to share why we did this and what our + experience was. TL;DR Our experience has convinced us that CockroachDB is + the best horizontally scalable database choice in 2020. + title: Choosing Cockroach DB for Horizontal Scalability +title: Choosing Cockroach DB for Horizontal Scalability + +--- + +Here at Storj Labs we just migrated our production databases from PostgreSQL to CockroachDB. We want to share why we did this and what our experience was. + +TL;DR Our experience has convinced us that CockroachDB is the best horizontally scalable database choice in 2020. + +# Why use a horizontally scalable database in the first place? + +Our top goal at Storj is to run the largest, most secure, decentralized, and distributed cloud storage platform. Our cloud storage holds its own against AWS S3 and Google Cloud Storage in performance and durability and also goes further by improving reliability since it's fully distributed. In order to compete on the same scale as the big cloud providers it's crucial we can scale our infrastructure. One of the ways we are doing this is by using a horizontally scalable database. To meet our first goal of storing an exabyte of data on the Storj network, the current architecture will store over 90 PBs of file metadata. Additionally, it's vital that the Storj Network can withstand multi-region failures and still keep the network up and the data available. All of this is made relatively easy with CockroachDB! + +# What about other horizontally scalable databases? + +We considered a number of different horizontally scalable databases, but for our needs, CockroachDB consistently came out on top. + +When considering a database that will horizontally scale there are three main categories of options: + +1. Shard your current relational database. +2. Use a NoSQL key-value database, like Cassandra or BigTable. +3. Use a NewSQL relational database, like Spanner or CockroachDB. + +Before the 2000s there weren’t any horizontally scaling database options. The only way to scale a database was to manually shard it yourself. Which is typically very tedious and kind of a nightmare. For example, it took Google over two years to shard some of their MySQL instances (1). Quite the undertaking! No wonder Google came up with the first “NewSQL” horizontally scalable relational database, Spanner. + +In the early 2000s, NoSQL became all the rage since they were the first horizontally scaling options. However, NoSQL has some tradeoffs, mainly weaker consistency guarantees, and no relational models. And here we are in the 2020s, finally what we always wanted, which is the rise of the strongly consistent, relational, horizontally scaling database. + +# What’s involved adding CockroachDB support to our application? + +Here is our process for getting CockroachDB running up with our application: + +1. Rewrote incompatible SQL code to be compatible with CockroachDB. +2. Performance and load tested in a QA environment. +3. Learned how to evaluate analytics about CockroachDB running in production. +4. Migrated production environments off PostgreSQL and over to CockroachDB. + +## Writing Compatible SQL + +One of the first parts of integrating with CockroachDB was to make sure all of our existing SQL was compatible. We were already backed by Postgres and CockroachDB is Postgresql wire protocol compatible, so we simply replaced the Postgres connection string with a CockroachDB connection URL and observed what broke. At the time (around v19.2-ish) there turned out to be quite a few PostgreSQL things that weren’t supported. Here’s a list of some of the highlights: + +* Primary keys must be created at table creation time, so you can’t delete and create them later on if you need to change it. +* No schemas, there’s only a single “public” schema. +* No stored procedures. +* No cursor support +* Additional incompatibilities can be found here: [https://www.cockroachlabs.com/docs/v20.1/postgresql-compatibility.html](https://www.cockroachlabs.com/docs/v20.1/postgresql-compatibility.html). + +Due to some of these unsupported Postgres features, we had to rewrite our migrations. Additionally, when we needed to change a primary key this resulted in a more tedious migration strategy where you create a new table with the new primary key then migrate all the data over to it and drop the old table. + +While this process to make our SQL compatible was a bit more tedious than I had hoped, it ended up taking about a month of full-time developer time, I still feel like it was much easier than migrating over to spanner or another database without postgres compatibility. Additionally since then, now in CockroachDB v20.1, many compatible issues have been resolved. CockroachDB is under fast development and is constantly listening to feedback from end-users and adding features per requests. + +## End-to-end testing, performance and load testing + +Once we had all the compatible SQL in place and all our unit tests passed, we then deployed to a production-like environment for some end-to-end testing and performance and load testing. Out of the box some things were faster than GCP CloudSQL Postgres, while some things were a teeny bit slower. + +## Performance Improvement Metrics + +One of the database access patterns we use is an ordered iterator, where we need to walk over every record in the database and perform some processing on each row. In our largest database with over six million rows, this iteration was getting very slow with CloudSQL Postgres database, taking about 13 hours, which was way too long. After we migrated to CockroachDB, processing every row in order was down to two hours! + +Additionally, we wrote a smaller script that emulated our production iterator access patterns, but in a more isolated coding environment and got the following results when performing processing on each row in order. CockroachDB was much faster. We think there are a few reasons for this improved performance, one being the data in the CockroachDB cluster is split across many nodes and therefore increases read throughput. + +``` +Speed of ordered iterator + +CockroachDB +Took 3.5s for 100,000 records +Took 18.8s for 1,000,000 records +Took 14m0.5s for 10,000,000 records + +CloudSQL Postgres +Took 56.8s for 100,000 records +Took 4m53.3s for 1000,000 records +Took 1h48m25.1s for 10,000,000 records +``` + +Another awesome feature of CockroachDB is prefix compression. CockroachDB data is stored in sorted order by the primary key and any prefix shared with the previous record is dropped (2). This saved a lot more space than we expected. While the data stored in CockroachDB is replicated three times (by default), the additional bytes on disk was just a little over two times Postgres since the prefix compression saved quite a bit of space. + +``` +Prefix compression: + +CloudSQL Postgres +239 GB +65,323,332 rows +~3658 bytes/row + +The same database ported to CockroachDB +186 GB +65,323,332 rows +~2846 bytes/row +``` + +## End-to-end Testing + +While end-to-end testing, there were three main issues we encountered: + +* Retry errors +* Transaction contention +* Stalled transactions that never completed + +### Retry Errors + +Everything in CockroachDB is run as a transaction, either an explicit transaction if the application code creates a transaction or CockroachDB will create an implicit transaction otherwise. If the transaction is implicit and fails, then CockroachDB will retry for you behind the scenes. However, if an explicit transaction is aborted/fails then it’s up to the application code to handle retries. For this, we added retry functionality to our database driver code [like so](https://github.com/storj/storj/blob/master/private/dbutil/cockroachutil/driver.go#L331). + +### Transaction Contention + +We experienced much more transaction contention with CockroachDB and therefore aborted transactions and also slower performance with some of our database access patterns. The following changes greatly improved these issues: + +* Use a smaller number of connections, fully saturated to help eliminate contention and improve throughput. This is especially helpful when there are many connections reading/writing from the same range. +* Multi-row upserts (reference [code](https://github.com/storj/storj/commit/955abd929304d41f51a651e1b6fabff2cac927b0)), instead of updating one row at a time, sends a single request with many upsert statements together. +* Blind writes to help reduce contention (reference [code](https://github.com/storj/storj/commit/78c6d5bb327185a5dfc4f3fd81f77f8f1310a180)). +* Bulk inserts. +* And more on CockroachDB [docs](https://www.cockroachlabs.com/docs/stable/performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). + +### Stalled Transactions + +We encountered some unusual behaviors where we had long-running queries taking over two hours sometimes. + +``` +-- run query from CockroachDB CLI to see age for active queries + +> SELECT node_id, + age(clock_timestamp(), oldest_query_start::timestamptz), + substring(active_queries, 0, 50) AS query +FROM [SHOW SESSIONS] +WHERE oldest_query_start IS NOT NULL +ORDER BY oldest_query_start ASC +LIMIT 10; + + node_id | age | query ++---------+----------------+----------------------------------------------+ + + 1 | 02:24:29.576152 | INSERT INTO example_table(id, first, last... + + 3 | 02:23:59.30406| INSERT INTO example_table(id, first, last... + + 2 | 02:23:51.504073 | INSERT INTO example_table(id, first, last... + + 1 | 02:23:35.517911 | INSERT INTO example_table(id, first, last... + + 3 | 02:23:21.543682 | INSERT INTO example_table(id, first, last... +``` + +Another way to see slow query times is: + +``` +SELECT * FROM [SHOW CLUSTER QUERIES] +WHERE start < (now() - INTERVAL '1 min'); +``` + +Luckily we run our databases on CockroachCloud, so the Cockroach Lab’s SREs and support team came to the rescue! They worked hard with us to debug this persistent stalled transaction issue that reared its head when we put the system under heavy load. It turned out that there was a recent change that merged into v19.2 where a potential downside is that UPDATE-heavy workloads experiencing high contention on many keys may have worse performance (up to O(n^2)). This seems to be the downside we hit. To solve this problem they deployed a patch to our cluster and now in v20.1 the fairness is adjusted when transactions conflict to reduce latencies under high contention. + +While this ended up being a one-off bug with the re-scheduling of transactions that CockroachDB has already implemented a fix for in v20.1, I think it’s an interesting experience to share because it displays a number of the reasons I love CockroachDB; .they work hard and fast to build a high-quality product and they consistently offer top-notch technical support. + +## Migrating the data during a maintenance window + +Once we had completed all of our testing we started migrating the actual data to CockroachDB. Luckily we were able to verify the migration steps and estimate how long it might take us for each of our deployments. We made the choice to make use of one of our maintenance windows which are Sunday 2 am-6 am eastern time. With the teams ready we began the migration steps. In order to ensure we didn’t miss any data, we had to stop all services within the deployment to stop all transactions. The next step was then to take a dump of all the data, with databases ranging in size from 110 GB to 260 GB. After waiting for the data dump to complete we then sorted the data to maximize import efficiency when importing to CockroachDB. The sorting steps took the longest, between 40 and 75 minutes. A small misjudgment on the size of these intermediate VMs meant that this step ended up taking significantly longer than we had estimated. With the sorting completed we then uploaded each snapshot to a cloud storage bucket and prepared our credentials to be passed to the CockroachDB servers to access the data dumps. The imports themselves took between 40 and 75 minutes as well. Once we had all the data imported we validated several tables to ensure the data had indeed been successfully imported, and then made the configuration changes for our application to be able to talk to its new database backend. + +## Adapting Existing Tooling to CockroachDB + +To ensure we have all of our bases covered we have a lot of automation around our deployments. One of these pieces is a backup step that ensures we have a snapshot backup of the database before we run any migrations. With the managed services for PostgreSQL that we’ve used in the past, we’ve always had an administrator API we can hit to trigger an automated backup. Cockroach Cloud already does a great job at running periodic backups as part of their managed service, but our requirements state that the backup we take before deployment is as close in time to the migration step as possible. + +We modified our existing scripts to now execute the CockroachDB backup commands directly in the database to trigger a full backup to one of our own storage buckets. Once these backups are complete we can then move forward with our existing deployment pipeline. + +1. [Google Spanner white paper](https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf) +2. RocksDB, underlying kv store of CockroachDB, [details](https://github.com/facebook/rocksdb/blob/master/table/block_based/block_builder.cc#L10) of prefix compression diff --git a/app/(blog)/blog/demystifying-technical-debt/bias.png b/app/(blog)/blog/demystifying-technical-debt/bias.png new file mode 100644 index 000000000..817fea5e6 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/bias.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/blackboard-diagram.png b/app/(blog)/blog/demystifying-technical-debt/blackboard-diagram.png new file mode 100644 index 000000000..e1738c93b Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/blackboard-diagram.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/borrowing.png b/app/(blog)/blog/demystifying-technical-debt/borrowing.png new file mode 100644 index 000000000..f0714f574 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/borrowing.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/code-pile.png b/app/(blog)/blog/demystifying-technical-debt/code-pile.png new file mode 100644 index 000000000..22fa19c04 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/code-pile.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/effort.png b/app/(blog)/blog/demystifying-technical-debt/effort.png new file mode 100644 index 000000000..67c00e8c9 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/effort.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/flowers-gopher.png b/app/(blog)/blog/demystifying-technical-debt/flowers-gopher.png new file mode 100644 index 000000000..c46f299c8 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/flowers-gopher.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/friends-with-the-monster.png b/app/(blog)/blog/demystifying-technical-debt/friends-with-the-monster.png new file mode 100644 index 000000000..82d775b86 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/friends-with-the-monster.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/hero.png b/app/(blog)/blog/demystifying-technical-debt/hero.png new file mode 100644 index 000000000..c9c3f4eef Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/hero.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/hype.png b/app/(blog)/blog/demystifying-technical-debt/hype.png new file mode 100644 index 000000000..7ac644b8b Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/hype.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/measuring-technical-debt.png b/app/(blog)/blog/demystifying-technical-debt/measuring-technical-debt.png new file mode 100644 index 000000000..9a62da432 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/measuring-technical-debt.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/one-plus-one.png b/app/(blog)/blog/demystifying-technical-debt/one-plus-one.png new file mode 100644 index 000000000..357c7fbc5 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/one-plus-one.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/page.md b/app/(blog)/blog/demystifying-technical-debt/page.md new file mode 100644 index 000000000..959e819a3 --- /dev/null +++ b/app/(blog)/blog/demystifying-technical-debt/page.md @@ -0,0 +1,267 @@ +--- +author: + name: Egon Elbre +date: '2021-10-14 00:00:00' +heroimage: ./hero.png +layout: blog +metadata: + description: "Technical debt has been bothering me for a while.\ + \ It looks like a scary monster in the closet. It seems somehow a catchall for\ + \ different design mistakes, code worsening over time, legacy codebases, and intentional\ + \ design mistakes due to time constraints." + title: Demystifying Technical Debt +title: Demystifying Technical Debt + +--- + +![](./technical-debt-monster.png) + +"Technical debt" has been bothering me for a while. It looks like a scary monster in the closet. It seems somehow a catchall for different design mistakes, code worsening over time, legacy codebases, and intentional design mistakes due to time constraints. You can take a look at the list of causes in [Wikipedia](https://en.wikipedia.org/wiki/Technical_debt#Causes) if you don't believe me. It makes you feel like the code is collecting dust when it's not being maintained, but clearly, that cannot be correct since the code might be unchanged. + +Let's take this piece of code from "Software Tools" by Kernighan and Plauger. It has been unchanged since 1976. Has the technical debt risen for this code? When we talk about things collecting dust, the book example would have more chance of being dusty than code stored digitally. + +![](./software-tools-book.jpeg) + +To push the metaphor to the breaking point, how do you measure technical debt, and how large is the interest? How much code would I need to write to pay off all the debt? If I have a lot of code, can I give a technical loan to other people? + +![](./measuring-technical-debt.png) + +But I digress; this unclear "technical debt" metaphor has caused bad decisions in codebases that don't need fixing. On the other hand, not understanding it has caused people to overlook actual problems. + +Before we get to tackle ***technical debt***, we need to take a slight detour. + + +## Quality and Effort + +The first problem we need to tackle is ***quality***. When we are talking about code quality, we usually have the following things in mind: + +* Fit for purpose - whether and how well the code does, what it is supposed to do +* Reliability - does it break every Tuesday between 1 AM - 2 AM; +* Security - can we access, modify or break information that isn't meant for us; +* Flexibility - how well can the code accommodate new needs; +* Efficiency - how many trees we need to burn to run an operation +* Maintainability - how many hours and lines of code do we need to modify to add, fix or remove a feature. + +![](./quality.png) + +When we talk about technical debt, usually, we are concerned about maintainability. There definitely are hints of the other aspects in there, but maintainability seems to be dominant. + +![](./effort.png) + +One way to summarize ***maintainability*** is to treat it as "***effort needed to make a change***." We can dissect this effort into several pieces or, in other words, places where we use up our energy: + +The most visible part is "***effort in code modification***." We can modify many different aspects of the code: + +* types, structs, variables, methods - the usual language primitives +* packages, modules - the grouping and organization of code +* tests - things that verify that the code works +* frontend, UX - how things look and how you interact with the system +* documentation - things that describe what the code or program does +* tooling, databases - changes to programs that the code needs to interact +* makefiles, build system - changes in how we run, build the code + +By no means is that list exhaustive. The less obvious part of the effort is "***effort in understanding***." Understanding here doesn't mean only ***understanding*** but clarifying and modifying things that help with understanding. We can dissect it into: + +* code structure - how clear is how things interact and how things are connected +* mental model - how we think about the problem and how it relates to the product +* product - how should the product work +* business value - how does the product give value to its users + +The last major category is about people. You rarely build a product alone. Even if you are the sole coder and owner of the company, you probably still need to communicate with your users. So, there's "***effort in communication***": + +* other developers - asking for help and discussing code design; +* code reviewers - giving and getting feedback on things that can be improved; +* product owners - discussing how the product should work; +* end-users - understanding their needs and where they would get the most value. + +We could dive deeper, but the main point is that ***effort*** is not one-dimensional and involves many human factors besides typing code. + +## Change in effort + +It's an obvious statement that this ***effort*** changes over time. The question is, how? + +***Code modification effort*** roughly depends on a few factors: the amount of code, code complexity, and understanding of the code. Based on these, we can estimate that effort to modify code usually increases because: + +* features are usually added -> more code and more things to understand; +* features are rarely removed -> amount of code doesn't decrease arbitrarily; +* the user interface is larger -> more things that can interact, hence more complexity; +* features accumulate more cases -> which means more complex and more code. + +![](./code-pile.png) + +***Understanding effort*** roughly depends on the complexity of the mental model, project, and user needs. It also depends on how much we know the system already. We can similarly estimate that it increases over time: + +* larger number of features interact -> more complex mental model and cases to consider; +* more business rules and concerns -> because we want to solve the user problems better; +* knowledge of code, that isn't being modified is forgotten -> it's going to be harder to work with a system that you don't know; +* people come and go -> tacit knowledge is lost when a person leaves. + +![](./blackboard-diagram.png) + +***Communication effort*** roughly depends on the number of people you need to communicate with and clarity on organization structure. Here it's harder to pinpoint clear tendencies, but we can estimate that: + +* communication effort increases when a company grows +* communication effort decreases when processes and company structure is clarified + +![](./sunny-gopher.png) + +Overall, we can estimate that: + +***The effort to maintain a project increases without activities that actively reduce it.*** + +![](./flowers-gopher.png) + +It would be easy to conclude that this "***increase in the effort***" is the "***technical debt***." However, when we look back at the initial question about old code. + +![](./software-tools-book.jpeg) + +This code has been years in a book without any new additions and no one communicating about it, but some still consider it technical debt. + +There must be things that we don't take into account when thinking about technical debt. + +## Mistakes everywhere + +One of the fundamental laws of software development is that you make mistakes. Technical debt is often associated with bad decisions in the past. Let's get philosophical – how do we recognize mistakes? + +![](./one-plus-one.png) + +When we look at this equation, we have two parts in our head: + +* The perception of the equation. +* The expectation of the equation and what it should be. + +Or in other words, there's something that we ***perceive*** and realize that it's not in its ***ideal*** state. The more significant this difference between our perception and expectation, the larger the mistake seems. + +We can apply the same line of thinking to ***effort*** and ***maintainability***. + +![](./perceived-vs-ideal-effort.png) + +Our ***ideal effort to modify*** decreases when we learn how things could be better. So, there's a "potential improvement" that we evaluate. We could simplify this into an equation: + +**Technical Debt ~ Perceived Effort - Ideal Effort** + +![](./potential-improvement.png) + +There are several interesting observations here. + +When there's a breakthrough in technology, people realize that there's a much better way to do something. Hence, they feel that their project has technical debt and they should fix it. Although, the effort to maintain the project hasn't changed. Only the expectation has changed. In principle, the technical debt is higher because people learned something new. *Note, that our "ideal" may have many problems that are being overlooked*. + +![](./hype.png) + +Borrowing technical debt is also nicely explained with this way of thinking. Instead of perceived effort and ideal effort changing separately, they are changed together. Or in other words, we increase perceived effort while knowing that ideal effort would increase it less. + +![](./borrowing.png) + +This model does seem to explain technical debt quite well and gives us a nice intuition about different situations. + +As a side note, it is interesting to consider ***quality debt*** or ***security debt***. However, it's essential to realize that improving ***quality*** can sometimes increase effort to maintain the software. For example, writing code is much easier if you don't care about security or performance. + +## Pobody's Nerfect + +It might seem that "***perceived effort***" and "***ideal effort***" are easy to measure, but they have many dimensions. Similarly, different people may come to different conclusions. + +The first question is, whose effort? – if we measure "hours spent on a change," then different people have different speeds. We could consider the "average developer in the world," or "average developer in the company," or "the average junior developer," or "average game developer." Additionally, people have different skills and knowledge in different areas. + +The second question is, which changes? Is it about an arbitrary change in the codebase or the most common change or architectural changes? All of these are different, and some are more likely than others. + +Finally, we need to consider the person evaluating because every person has some biases. Especially when dealing with "***perceived effort***" and "***ideal effort***." + +![](./bias.png) + +For example, if the person is hyped about a language, framework, or works with a system they know well, they can easily underestimate the average effort. This is due to knowing how to solve common problems and knowing how to avoid the issues in the first place. + +On the other hand, if the person has a strong preference for other tools, the tools have a flaky behavior, or the person doesn't understand the system, they can overestimate the effort needed to maintain a system. For example, flaky tests that fail once a month are annoying; however, realistically, it doesn't affect the effort to maintain too much. + +We tend to overestimate the effort needed to maintain code written in old languages. Definitely, there is more effort required to maintain old code, but it's not as big as it might seem. Think about how people learn a new JavaScript framework and library every week and keep up with it. If you can learn new code, you can learn old code. + +We also tend to overestimate the effort needed to use another language with different features. A C++ programmer starting to use Go would feel overly restricted and hence conclude that they will be significantly slower when writing. Similarly, a Go programmer thinks they would be overwhelmed when starting to use Rust due to the number of available features. Both are right to some degree, but the main reason for feeling the "speed of writing difference" is not knowing how to use the language effectively. After a few months of using a language, the unfamiliarity will decrease. There are definitely differences in languages and their usability, but it's not as big as it seems at first sight. Nevertheless, there would still be a bias towards the language and community you like more. + +Interestingly there's no such feeling when taking a language with an unfamiliar paradigm. In such cases, we accept our ignorance much more quickly. + +Beginner programmers seem to overestimate the "ideal effort" for newer frameworks because it might look like they solve all the problems. Veteran programmers are either realistic or pessimistic because they have been burnt before and know that most problems don't lie in the framework but elsewhere. + +Overall we can state that the less familiar you are with a codebase, system, tool, the higher your bias can be. The bias can be either positive or negative. + +## Technical Debt by Ward Cunningham + +Initially, when Ward Cunningham came up with the metaphor, he only had the "code mismatching business ideas" in mind. He was more precise in its formulation than people know. + +> And that said that if we failed to make our program align with what we then understood to be the proper way to think about our financial objects, then we were gonna continually stumble over that disagreement and that would slow us down which was like paying interest on a loan. +> +> Ward Cunningham, [wiki](http://wiki.c2.com/?WardExplainsDebtMetaphor) + +{% youtube-embed videoId="pqeJFYwnkjE" /%} + +In other words, we improve our "ideal mental model of the system," and there's a difference between our code and the ideal mental model. There was no concept of "borrowing," and that would've been an error while developing. + +## What can you do about it? + +After all of this discussion, you might wonder how you deal with technical debt. + +### Rewrite??? + +The first inclination for people to get rid of "technical debt" is to rewrite the system. Rewriting carries considerable risk, and the larger the piece you are rewriting, the larger the chance of failure. + +Few factors contribute to rewriting ending up in a failure: + +* People don't notice things that work well in the current system due to desensitization. During rewriting, it's easy to forget that they should keep working well. These parts are also often more important than the things that currently don't work well. +* *Note, people also get desensitized to things that work poorly consistently. For example, when some process always takes 10 min, it doesn't bother people; however, when it takes 10 min randomly, it does.* +* Size of the refactoring. Each line of code you change can introduce a bug; hence, the more lines and more systems the piece of code integrates, the more likely it is to have faults. +* People focus on fixing the mistakes, sometimes at the cost of the rest of the system. One aspect of this is the "second-system effect," where you end up overcomplicating the system by including all the missing features. +* Unclear understanding of how the current system exactly works. It's pretty common that people want to rewrite a system because they don't understand it. + +Overall, a [rewrite should be the last resort](https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/), and try to minimize the problems above to ensure that the rewrite ends up as a success. Before rewriting, you also should be able to give an estimate (in numbers) on how much the rewrite would help. + +Prefer refactoring over rewriting and see "Working Effectively with Legacy Code" by Michael C. Feathers for more recommendations. + +### Continuous Learning + +One good way to prevent "technical debt" is to ensure that the developers have a scheduled time to learn about coding and the business. The more the developers know how to do things, the fewer surprises they get about their system. There are many ways to implement these in a company - 20% projects, hack weeks, book clubs, regular presentations by other people. + +Try to implement one feature in multiple ways. The more ways you know how to solve a problem, the more informed your decision will be. + +Finally, a good strategy is to ask for help and guidance. There are plenty of experienced programmers that can do a review of your code and suggest improvements. + +When we learn new things, it'll actually end up increasing "technical debt" because it lowers the "ideal effort,"… but it will also mean that programmers are more likely to write code that is nearer to that "ideal effort." + +### Code Reviews + +Code reviews significantly help disseminate understanding about the system and learn early about things that could be better. + +First, automate linting as much as possible and automate formatting. Ideally, when someone starts to review the code, all the style questions have already been solved. The style questions can grab attention quite fast. + +One target for every developer is to ensure your next pull request quality is better than the last one. By trying to improve the quality of every PR, people end up defaulting to a better baseline. There are many ways to define better; ideally, try to improve in multiple dimensions. + +Strive for gradual improvement of the codebase rather than large improvements. As mentioned previously, the larger the change, the more likely it is to contain mistakes. + +Ideally, target less than 400 LOC per change. When the change is over 400 LOC, the reviewer fatigue kicks in, and reviewers start missing more bugs. Similarly, when commits are small, they get merged faster and are less likely to go stale. [See this SmartBear study for more information](https://smartbear.com/learn/code-review/best-practices-for-peer-code-review/). + +While reviewing, always consider whether two similar commits would make it difficult to maintain? If yes, then the next PR should be preceded by an improvement to the structure. + +### Maintenance + +It's easy to lose sight of the overall picture when implementing things one at a time. Hence, do regular architecture reviews. Think about whether everything feels right. Note down places where people waste their effort and discuss how you can improve these parts. + +As programmers, the first inclination is to fix "maintenance effort" with fixing the code; there sometimes can be alternative means. For example, a good video explaining how the existing "bad system" works and why things ended up that way can be much less effort and have more impact. + +For maintenance, it's helpful to isolate problem areas. Some third-party packages and libraries are pervasive and can seriously affect the rest of the codebase. By creating a nice wrapper for those systems, they can be made less benign. + +### Acceptance + +The final advice is about acceptance: + +> Not all of a large system will be well designed... +> +> Eric Evans + +While the inclination is to try to fix all the problems you notice, it might not make a significant difference to the needed effort to maintain the system. You don't have to rewrite your bash scripts in Haskell for the glory of purity. Your time to implement things is limited and try to figure out how you can make the most impact on the value stream. + +## Conclusion + +Technical debt is not "dust" that accumulates on your code, but rather it's an inherent part of code. Over time you learn and notice mistakes in your systems. Using "technical debt accumulates" is the wrong mentality; instead, it should be considered "discovering technical debt." + +> Technical Debt is not a Monster. +> +> It's just the realization that you could do better. + +![](./friends-with-the-monster.png) diff --git a/app/(blog)/blog/demystifying-technical-debt/perceived-vs-ideal-effort.png b/app/(blog)/blog/demystifying-technical-debt/perceived-vs-ideal-effort.png new file mode 100644 index 000000000..c8c48d89e Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/perceived-vs-ideal-effort.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/potential-improvement.png b/app/(blog)/blog/demystifying-technical-debt/potential-improvement.png new file mode 100644 index 000000000..a9be196d5 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/potential-improvement.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/quality.png b/app/(blog)/blog/demystifying-technical-debt/quality.png new file mode 100644 index 000000000..c0322c3c1 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/quality.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/software-tools-book.jpeg b/app/(blog)/blog/demystifying-technical-debt/software-tools-book.jpeg new file mode 100644 index 000000000..64049a6be Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/software-tools-book.jpeg differ diff --git a/app/(blog)/blog/demystifying-technical-debt/sunny-gopher.png b/app/(blog)/blog/demystifying-technical-debt/sunny-gopher.png new file mode 100644 index 000000000..6178630a7 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/sunny-gopher.png differ diff --git a/app/(blog)/blog/demystifying-technical-debt/technical-debt-monster.png b/app/(blog)/blog/demystifying-technical-debt/technical-debt-monster.png new file mode 100644 index 000000000..891ec1809 Binary files /dev/null and b/app/(blog)/blog/demystifying-technical-debt/technical-debt-monster.png differ diff --git a/app/(blog)/blog/finding-and-tracking-resource-leaks-in-go/hero.png b/app/(blog)/blog/finding-and-tracking-resource-leaks-in-go/hero.png new file mode 100644 index 000000000..bbd0eae58 Binary files /dev/null and b/app/(blog)/blog/finding-and-tracking-resource-leaks-in-go/hero.png differ diff --git a/app/(blog)/blog/finding-and-tracking-resource-leaks-in-go/page.md b/app/(blog)/blog/finding-and-tracking-resource-leaks-in-go/page.md new file mode 100644 index 000000000..932d6eabf --- /dev/null +++ b/app/(blog)/blog/finding-and-tracking-resource-leaks-in-go/page.md @@ -0,0 +1,402 @@ +--- +author: + name: Egon Elbre +date: '2022-10-13 00:00:00' +heroimage: ./hero.png +layout: blog +metadata: + description: Forgetting to close a file, a connection, or some other resource is + a rather common issue in Go. Usually you can spot them with good code review practices, + but what if you wanted to automate it and you don't have a suitable linter at + hand? + title: Finding and Tracking Resource Leaks in Go +title: Finding and Tracking Resource Leaks in Go + +--- + +Forgetting to close a file, a connection, or some other resource is a rather common issue in Go. Usually you can spot them with good code review practices, but what if you wanted to automate it and you don't have a suitable linter at hand? + +How do we track and figure out those leaks? + +Fortunately, there's an approach to finding common resource leaks that we’ll explore below. + +## Problem: Connection Leak + +Let's take a simple example that involves a TCP client. Of course, it applies to other protocols, such as GRPC, database, or HTTP. We'll omit the communication implementation because it's irrelevant to the problem. + +```go +type Client struct { + conn net.Conn +} + +func Dial(ctx context.Context, address string) (*Client, error) { + conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", address) + if err != nil { + return nil, fmt.Errorf("failed to dial: %w", err) + } + + return &Client{conn: conn}, nil +} + +func (client *Client) Close() error { + return client.conn.Close() +} +``` + +It's easy to put the defer in the wrong place or forget to call Close altogether. + +```go +func ExampleDial(ctx context.Context) error { + source, err := Dial(ctx, "127.0.0.1:1000") + if err != nil { + return err + } + + destination, err := Dial(ctx, "127.0.0.1:1001") + if err != nil { + return err + } + + defer source.Close() + defer destination.Close() + + data, err := source.Recv(ctx) + if err != nil { + return fmt.Errorf("recv failed: %w", err) + } + + err = destination.Send(ctx, data) + if err != nil { + return fmt.Errorf("send failed: %w", err) + } + + return nil +} +``` + +Notice if we fail to dial the second client, we have forgotten to close the source connection. + +## Problem: File Leak + +Another common resource management mistake is a file leak. + +```go +func ExampleFile(ctx context.Context, fs fs.FS) error { + file, err := fs.Open("data.csv") + if err != nil { + return fmt.Errorf("open failed: %w", err) + } + + stat, err := fs.Stat() + if err != nil { + return fmt.Errorf("stat failed: %w", err) + } + + fmt.Println(stat.Name()) + + _ = file.Close() + return nil +} +``` + +## Tracking Resources + +How do we track and figure out those leaks? One thing we can do is to keep track of every single open file and connection and ensure that everything is closed when the tests finish. + +We need to build something that keeps a list of all open things and tracks where we started using a resource. + +To figure out where our "leak" comes from, we can use [`runtime.Callers`](https://pkg.go.dev/runtime#Callers). You can look at the [Frames example](https://pkg.go.dev/runtime#example-Frames) to learn how to use it. Let's call the struct we use to hold this information a `Tag`. + +```go +// Tag is used to keep track of things we consider open. +type Tag struct { + owner *Tracker // we'll explain this below + caller [5]uintptr +} + +// newTag creates a new tracking tag. +func newTag(owner *Tracker, skip int) *Tag { + tag := &Tag{owner: owner} + runtime.Callers(skip+1, tag.caller[:]) + return tag +} + +// String converts a caller frames to a string. +func (tag *Tag) String() string { + var s strings.Builder + frames := runtime.CallersFrames(tag.caller[:]) + for { + frame, more := frames.Next() + if strings.Contains(frame.File, "runtime/") { + break + } + fmt.Fprintf(&s, "%s\n", frame.Function) + fmt.Fprintf(&s, "\t%s:%d\n", frame.File, frame.Line) + if !more { + break + } + } + return s.String() +} + +// Close marks the tag as being properly deallocated. +func (tag *Tag) Close() { + tag.owner.Remove(tag) +} +``` + +Of course, we need something to keep the list of all open trackers: + +```go +// Tracker keeps track of all open tags. +type Tracker struct { + mu sync.Mutex + closed bool + open map[*Tag]struct{} +} + +// NewTracker creates an empty tracker. +func NewTracker() *Tracker { + return &Tracker{open: map[*Tag]struct{}{}} +} + +// Create creates a new tag, which needs to be closed. +func (tracker *Tracker) Create() *Tag { + tag := newTag(tracker, 2) + + tracker.mu.Lock() + defer tracker.mu.Unlock() + + // We don't want to allow creating a new tag, when we stopped tracking. + if tracker.closed { + panic("creating a tag after tracker has been closed") + } + tracker.open[tag] = struct{}{} + + return tag +} + +// Remove stops tracking tag. +func (tracker *Tracker) Remove(tag *Tag) { + tracker.mu.Lock() + defer tracker.mu.Unlock() + delete(tracker.open, tag) +} + +// Close checks that none of the tags are still open. +func (tracker *Tracker) Close() error { + tracker.mu.Lock() + defer tracker.mu.Unlock() + + tracker.closed = true + if len(tracker.open) > 0 { + return errors.New(tracker.openResources()) + } + return nil +} + +// openResources returns a string describing all the open resources. +func (tracker *Tracker) openResources() string { + var s strings.Builder + fmt.Fprintf(&s, "%d open resources\n", len(tracker.open)) + + for tag := range tracker.open { + fmt.Fprintf(&s, "---\n%s\n", tag) + } + + return s.String() +} +``` + +Let's look at how it works: + +```go +func TestTracker(t *testing.T) { + tracker := NewTracker() + defer func() { + if err := tracker.Close(); err != nil { + t.Fatal(err) + } + }() + + tag := tracker.Create() + // if we forget to call Close, then the test fails. + // tag.Close() +} +``` + +You can test it over at https://go.dev/play/p/8AkKrzYVFH5. + +## Hooking up the tracker to a `fs.FS` + +We need to integrate it into the initially problematic code. We can create a wrapper for `fs.FS` that creates a tag for each opened file. + +```go +type TrackedFS struct { + tracker *Tracker + fs fs.FS +} + +func TrackFS(fs fs.FS) *TrackedFS { + return &TrackedFS{ + tracker: NewTracker(), + fs: fs, + } +} + +func (fs *TrackedFS) Open(name string) (fs.File, error) { + file, err := fs.fs.Open(name) + if err != nil { + return file, err + } + + tag := fs.tracker.Create() + return &trackedFile{ + File: file, + tag: tag, + }, nil +} + +func (fs *TrackedFS) Close() error { return fs.tracker.Close() } + +type trackedFile struct { + fs.File + tag *Tag +} + +func (file *trackedFile) Close() error { + file.tag.Close() + return file.File.Close() +} +``` + +Finally, we can use this wrapper in a test and get some actual issues resolved: + +```go +func TestFS(t *testing.T) { + // We'll use `fstest` package here, but you can also replace this with + // `os.DirFS` or similar. + dir := fstest.MapFS{ + "data.csv": &fstest.MapFile{Data: []byte("hello")}, + } + + fs := TrackFS(dir) + defer func() { + if err := fs.Close(); err != nil { + t.Fatal(err) + } + }() + + file, err := fs.Open("data.csv") + if err != nil { + t.Fatal(err) + } + + stat, err := file.Stat() + if err != nil { + t.Fatal(err) + } + + t.Log(stat.Name()) +} +``` + +You can play around with it here https://go.dev/play/p/VTKZUzWukTe. + + +## Hooking up the tracker via a `Context` + +Passing this `tracker` everywhere would be rather cumbersome. However, we can write some helpers to put the tracker inside a `Context`. + +```go +type trackerKey struct{} + +func WithTracker(ctx context.Context) (*Tracker, context.Context) { + tracker := NewTracker() + return tracker, context.WithValue(ctx, trackerKey{}, tracker) +} + +func TrackerFromContext(ctx context.Context) *Tracker { + value := ctx.Value(trackerKey{}) + return value.(*Tracker) +} +``` + +Of course, we need to adjust our `Client` implementation as well: + +```go +type Client struct { + conn net.Conn + tag *Tag +} + +func Dial(ctx context.Context, address string) (*Client, error) { + conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", address) + if err != nil { + return nil, fmt.Errorf("failed to dial: %w", err) + } + + tracker := TrackerFromContext(ctx) + return &Client{conn: conn, tag: tracker.Create()}, nil +} + +func (client *Client) Close() error { + client.tag.Close() + return client.conn.Close() +} +``` + +To make our testing code even shorter, we can make a tiny helper: + +```go +func TestingTracker(ctx context.Context, tb testing.TB) context.Context { + tracker, ctx := WithTracker(ctx) + tb.Cleanup(func() { + if err := tracker.Close(); err != nil { + tb.Fatal(err) + } + }) + return ctx +} +``` + +Finally, we can put it all together: + +```go +func TestClient(t *testing.T) { + ctx := TestingTracker(context.Background(), t) + + addr := startTestServer(t) + + client, err := Dial(ctx, addr) + if err != nil { + t.Fatal(err) + } + + // if we forget to close, then the test will fail + // client.Close + _ = client +} +``` + +You can see it working over here https://go.dev/play/p/B6qI6xgij1m. + +## Making it zero cost for production + +Now, all of this `runtime.Callers` calling comes with a high cost. However, we can reduce it by conditionally compiling the code. Luckily we can use tags to only compile it only for testing. I like to use the `race` tag for it because it is added any time you run your tests with `-race`. + +```go +//go:build race + +package tracker +``` + +The implementations are left as an exercise for the reader. :) + +## Conclusion + +This is probably not a final solution for your problem, but hopefully, it is a good starting point. You can add more helpers, maybe track the filename inside a `Tag`, or only print unique caller frames in the test failure. Maybe try implementing this for SQL driver and track each thing separately -- you can take a peek [at our implementation](https://github.com/storj/private/tree/main/tagsql), if you get stuck. + +May all your resource leaks be discovered. + +This is a continuation of our series of finding leaks in Golang. In case you missed it, in a previous post we covered [finding leaked goroutines](https://www.storj.io/blog/finding-goroutine-leaks-in-tests). diff --git a/app/(blog)/blog/finding-goroutine-leaks-in-tests/hero.jpeg b/app/(blog)/blog/finding-goroutine-leaks-in-tests/hero.jpeg new file mode 100644 index 000000000..d78c9d33b Binary files /dev/null and b/app/(blog)/blog/finding-goroutine-leaks-in-tests/hero.jpeg differ diff --git a/app/(blog)/blog/finding-goroutine-leaks-in-tests/page.md b/app/(blog)/blog/finding-goroutine-leaks-in-tests/page.md new file mode 100644 index 000000000..e0787e849 --- /dev/null +++ b/app/(blog)/blog/finding-goroutine-leaks-in-tests/page.md @@ -0,0 +1,297 @@ +--- +author: + name: Egon Elbre +date: '2022-03-07 00:00:00' +heroimage: ./hero.jpeg +layout: blog +metadata: + description: 'A leaked goroutine at the end of a + test can indicate several problems. Let''s first, take a look at the most common + ones before tackling an approach to finding them.Problem: DeadlockFirst, we can + have a goroutine that is blocked. As an example:func LeakySumSquares(c...' + title: Finding Goroutine Leaks in Tests +title: Finding Goroutine Leaks in Tests + +--- + +A leaked goroutine at the end of a test can indicate several problems. Let's first, take a look at the most common ones before tackling an approach to finding them. + +### Problem: Deadlock + +First, we can have a goroutine that is blocked. As an example: + + +```go +func LeakySumSquares(ctx context.Context, data []int) ( + total int, err error) { + + results := make(chan int) + + for _, v := range data { + v := v + go func() { + result := v * v + results <- result + }() + } + + for { + select { + case value := <-results: + total += value + case <-ctx.Done(): + return ctx.Err() + } + } + + return total, nil +} + +``` +In this case, when the context is canceled, the goroutines might end up leaking. + +### Problem: Leaked Resource + +Many times different services, connections, or databases have an internal goroutine used for async processing. A leaked goroutine can show such leaks. + + +```go +type Conn struct { + messages chan Message + + close context.CancelFunc + done chan struct{} +} + +func Dial(ctx context.Context) *Conn { + ctx, cancel := context.WithCancel(ctx) + conn := &Conn{ + close: cancel, + messages: make(chan Message) + done: make(chan struct{}), + } + go conn.monitor(ctx) + return conn +} + +func (conn *Conn) monitor(ctx context.Context) { + defer close(conn.done) + for { + select { + case msg := <-conn.messages: + conn.handle(msg) + case <-ctx.Done(): + return + } + } +} + +func (conn *Conn) Close() { + conn.close() + <-conn.done +} + +``` +Even if the main loop is properly handled, the *conn.handle(msg)* could become deadlocked in other ways. + + +### Problem: Lazy Closing Order + + +Even if all the goroutines terminate, there can still be order problems with regard to resource usage. For example, you could end up depending on a database, connection, file, or any other resource, that gets closed before the goroutine finishes. + + +Let's take a common case of the problem: + + +```go +type Server struct { + log Logger + db *sql.DB +} + +func NewServer(log Logger, dburi string) (*Server, error) { + db, err := sql.Open("postgres", dburi) + if err != nil { + return nil, fmt.Errorf("opening database failed: %w", err) + } + return &Server{log: log, db: db}, nil +} + + +func (server *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { + tag := r.FormValue("tag") + if tag == "" { + return + } + + // update the database in the background + go func() { + err := server.db.Exec("...", tag) + if err != nil { + server.log.Errorf("failed to update tags: %w", err) + } + }() +} + + +func (server *Server) Close() { + _ = server.db.Close() +} + +``` +In this case, when the *Server* is closed, there still could be goroutines updating the database in the background. Similarly, even the *Logger* could be closed before the goroutine finishes, causing some other problems. + + +The severity of such close ordering depends on the context. Sometimes it's a simple extra error in the log; in other cases, it can be a data-race or a panic taking the whole process down. + +### Rule of Thumb + +Hopefully, it's clear that such goroutines can be problematic. + +One of the best rules in terms of preventing these issues is: + + +The location that starts the goroutine must wait for the goroutine to complete even in the presence of context cancellation. Or, it must explicitly transfer that responsibility to some other service. + +As long as you close the top-level service responsible for everything, it'll become visible in tests because if there's a leak, then the test cannot finish. + +Unfortunately, this rule cannot be applied to third-party libraries and it's easy to forget to add tracking to a goroutine. + + +### Finding Leaks + +We could use the total number of goroutines, to find leaks at the end of a test, however that wouldn't work with parallel tests. + + +One helpful feature in Go is [goroutine labels](https://rakyll.org/profiler-labels/), which can make profiling and stack traces more readable. One interesting feature they have is that they are propagated automatically to child goroutines. + + +This means if we attach a unique label to a goroutine, we should be able to find all the child goroutines. However, code for finding such goroutines is not trivial. + + +To attach the label: + + + +```go +func Track(ctx context.Context, t *testing.T, fn func(context.Context)) { + label := t.Name() + pprof.Do(ctx, pprof.Labels("test", label), fn) + if err := CheckNoGoroutines("test", label); err != nil { + t.Fatal("Leaked goroutines\n", err) + } +} + +``` +Unfortunately, currently, there's not an easy way to get the goroutines with a given label. But, we can use some of the profiling endpoints to extract the necessary information. Clearly, this is not very efficient. + + +```go +import "github.com/google/pprof/profile" + +func CheckNoGoroutines(key, value string) error { + var pb bytes.Buffer + profiler := pprof.Lookup("goroutine") + if profiler == nil { + return fmt.Errorf("unable to find profile") + } + err := profiler.WriteTo(&pb, 0) + if err != nil { + return fmt.Errorf("unable to read profile: %w", err) + } + + p, err := profile.ParseData(pb.Bytes()) + if err != nil { + return fmt.Errorf("unable to parse profile: %w", err) + } + + return summarizeGoroutines(p, key, value) +} + +func summarizeGoroutines(p *profile.Profile, key, expectedValue string) ( + err error) { + var b strings.Builder + + for _, sample := range p.Sample { + if !matchesLabel(sample, key, expectedValue) { + continue + } + + fmt.Fprintf(&b, "count %d @", sample.Value[0]) + // format the stack trace for each goroutine + for _, loc := range sample.Location { + for i, ln := range loc.Line { + if i == 0 { + fmt.Fprintf(&b, "# %#8x", loc.Address) + if loc.IsFolded { + fmt.Fprint(&b, " [F]") + } + } else { + fmt.Fprint(&b, "# ") + } + if fn := ln.Function; fn != nil { + fmt.Fprintf(&b, " %-50s %s:%d", fn.Name, fn.Filename, ln.Line) + } else { + fmt.Fprintf(&b, " ???") + } + fmt.Fprintf(&b, "\n") + } + } + fmt.Fprintf(&b, "\n") + } + + if b.Len() == 0 { + return nil + } + + return errors.New(b.String()) +} + +func matchesLabel(sample *profile.Sample, key, expectedValue string) bool { + values, hasLabel := sample.Label[key] + if !hasLabel { + return false + } + + for _, value := range values { + if value == expectedValue { + return true + } + } + + return false +} + +``` +And a failing test might look like this: + + +```go +func TestLeaking(t *testing.T) { + t.Parallel() + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + Track(ctx, t, func(ctx context.Context) { + LeakyThing(ctx) + }) +} + +func LeakyThing(ctx context.Context) { + done := make(chan struct{}) + go func() { + go func() { + done <- struct{}{} + }() + done <- struct{}{} + }() +} + +``` +The full example can be found here . + +Depending on your use case, you may want to adjust to your needs. For example, you may want to skip some goroutines or maybe print some extra information, or have a grace period for transient goroutines to shut down. + +Such an approach can be hooked into your tests or existing system in a multitude of ways. + diff --git a/app/(blog)/blog/flexible-file-sharing-with-macaroons/1597845bc69c4f76.png b/app/(blog)/blog/flexible-file-sharing-with-macaroons/1597845bc69c4f76.png new file mode 100644 index 000000000..3c44ee859 Binary files /dev/null and b/app/(blog)/blog/flexible-file-sharing-with-macaroons/1597845bc69c4f76.png differ diff --git a/app/(blog)/blog/flexible-file-sharing-with-macaroons/441ae738c61703c8.png b/app/(blog)/blog/flexible-file-sharing-with-macaroons/441ae738c61703c8.png new file mode 100644 index 000000000..02f2fcca7 Binary files /dev/null and b/app/(blog)/blog/flexible-file-sharing-with-macaroons/441ae738c61703c8.png differ diff --git a/app/(blog)/blog/flexible-file-sharing-with-macaroons/71499072b0db295f.jpeg b/app/(blog)/blog/flexible-file-sharing-with-macaroons/71499072b0db295f.jpeg new file mode 100644 index 000000000..0b4a45746 Binary files /dev/null and b/app/(blog)/blog/flexible-file-sharing-with-macaroons/71499072b0db295f.jpeg differ diff --git a/app/(blog)/blog/flexible-file-sharing-with-macaroons/79fa81cab0393039.png b/app/(blog)/blog/flexible-file-sharing-with-macaroons/79fa81cab0393039.png new file mode 100644 index 000000000..5bdf1ec89 Binary files /dev/null and b/app/(blog)/blog/flexible-file-sharing-with-macaroons/79fa81cab0393039.png differ diff --git a/app/(blog)/blog/flexible-file-sharing-with-macaroons/page.md b/app/(blog)/blog/flexible-file-sharing-with-macaroons/page.md new file mode 100644 index 000000000..c5539565a --- /dev/null +++ b/app/(blog)/blog/flexible-file-sharing-with-macaroons/page.md @@ -0,0 +1,41 @@ +--- +author: + name: Paul Cannon +date: '2019-05-03 00:00:00' +heroimage: ./71499072b0db295f.jpeg +layout: blog +metadata: + description: Sharing is a vital piece of any online storage system. Or, to be more + precise, access control is a vital piece of such systems. When you store a file, + you need to be able to designate whether other people or automated agents are + allowed to retrieve the data, delete it, or put something else in it... + title: Flexible File Sharing With Macaroons +title: Flexible File Sharing With Macaroons + +--- + +Sharing is a vital piece of any online storage system. Or, to be more precise, access control is a vital piece of such systems. When you store a file, you need to be able to designate whether other people or automated agents are allowed to retrieve the data, delete it, or put something else in its place. On top of that, you also need to be able to designate when and for how long that particular access should be allowed. We can refer to this as “sharing” because that is typically how we make direct use of access control in our everyday online lives. You might have shared a Google Docs spreadsheet with a team of people or allowed a friend to download a particular video file from your Dropbox account. Often, this type of sharing involves sending around a unique URL. + +The Storj platform, of course, requires this type of functionality. Some people will be using Storj as personal backup, and won’t want or need anyone else to have access to their stuff. Some will use the platform to store collections of family photos, and they will want to allow friends and family to view the albums (but not add to them or change them). Some will use Storj for business collaboration, and share a large folder with members of a team as though it were a network share. Finally, some will use Storj as a CDN, and will want to allow anyone and everyone to download their content. Storj intends to support all these use cases and many more. + +The mechanism we are building to accomplish all of this uses a special type of bearer credential called a Macaroon. Macaroons can be thought of as “enhanced [browser] cookies”, and that’s why they got the delicious name. They were an idea that originated from Google research¹ a few years ago. Macaroons have the special quality that the user can derive new, more restricted macaroons from them, without any involvement on the part of the service they are targeting. This is useful to us for two chief reasons. First, because macaroons encode all the necessary information about what they grant, Storj Satellites do not need to maintain their own (potentially enormous) list of all shares ever made, including what resources are included, with whom they are shared, and under what conditions access should be allowed. Instead, the Satellites only² need to keep track of one root token per project. Any derived Macaroons that are sent to the Satellite can be parsed and verified when received. The second reason is that this mechanism allows third-party developers a significant amount of power and flexibility, without requiring the development of a powerful and complicated API, along with all the costs and complexity and potential vulnerabilities that would entail. + +Let’s see how they work! + +Suppose you create a new project instance on the Storj network called “OlioFamilyPhotos”. The Satellite you are using gives you a token that will govern access to that project. We can think of it looking like a certificate like this: + +![Macaroons are tasty!](./441ae738c61703c8.png)Any time you ask your computer to read or write files in OlioFamilyPhotos, it can send that special certificate to the Storj Satellite to prove that it’s allowed to do so. The Satellite will verify the digital signature and grant access if appropriate. You could make a copy of the certificate and share it with your spouse, if you trust them with full, unrestricted access to the project. + +But you may want to allow other family members to see these photos, without being able to reorganize or delete anything (they really are busybodies sometimes). Rather than making API calls to the Storj Satellite to ask for a new token, you³ can make a copy of your existing token, cut off the signature, paste it into a bigger certificate, and add on a proviso, like this: + +![But while they are good,](./79fa81cab0393039.png)You can hand out copies of this bigger certificate to family members at the next family reunion. If they want to see your photos, their computers will send this bigger certificate to the Satellite. The Satellite can still verify the digital signature—through the wonders of digital cryptography⁴—and thereby verify that the added proviso is satisfied. If your second cousin-in-law turns out to be nefarious and wants to make changes to your photos, they can’t just cut out the smaller certificate from the big one and use that, because its signature is gone. They could try to make a new bigger certificate with a weaker proviso, but they would not be able to make a valid signature because they don’t know what the original signature looked like. + +Now, imagine Aunt Alice wants to share a specific one of your photos with her friend Beth. Alice values your privacy and does not want to share everything with Beth, and she also does not want Beth to be able to share the photo with anyone else. Just like what we did earlier, Alice can make a copy of her certificate, cut off the signature, paste it into a bigger certificate, and add on some provisos: + +![Macaroons are not as good as macarons!](./1597845bc69c4f76.png) + +Again, the Satellite will be able to verify this digital signature, verify that the signer knew what the signature on the intermediate certificate looked like, verify that the intermediate signature was also valid, and check that all the provisos there are satisfied before granting access. There won’t be any way for Beth to use the original root project token or the intermediate family-sharing token on their own; she will only be able to use this certificate to fetch that one specific file, and nothing else. She also won’t be able to pass on her access to anyone else, because of the “only if the bearer is Beth” proviso that has been indelibly added. + +This chain can be continued for many more steps, allowing Macaroons of significant complexity where necessary. + +We expect to bring you access control by way of Macaroons in one of the next few Alpha releases. Stay tuned for more details! diff --git a/app/(blog)/blog/go-integration-tests-with-postgres/hero.jpeg b/app/(blog)/blog/go-integration-tests-with-postgres/hero.jpeg new file mode 100644 index 000000000..c3fc2ef12 Binary files /dev/null and b/app/(blog)/blog/go-integration-tests-with-postgres/hero.jpeg differ diff --git a/app/(blog)/blog/go-integration-tests-with-postgres/page.md b/app/(blog)/blog/go-integration-tests-with-postgres/page.md new file mode 100644 index 000000000..96a2a5c11 --- /dev/null +++ b/app/(blog)/blog/go-integration-tests-with-postgres/page.md @@ -0,0 +1,394 @@ +--- +author: + name: Egon Elbre +date: '2023-03-20 00:00:00' +heroimage: ./hero.jpeg +layout: blog +metadata: + description: When writing server side projects in Go, at some point you will also + need to test against a database. Let's take a look at different ways of using + Postgres with different performance characteristics. The final approach shows + how you can set up a clean database in 20ms (there are a few caveats). + title: Go Integration Tests with Postgres +title: Go Integration Tests with Postgres + +--- + +When writing server side projects in Go, at some point you will also need to test against a database. Let's take a look at different ways of using Postgres with different performance characteristics. The final approach shows how you can set up a clean database in 20ms (there are a few caveats). + +We're not going to cover the "how should you use a real database in your tests" debate. At some point you'll need to test your database layer, so, we'll cover those cases. + +## Using containers + +If you have searched a bit on how to set up a clean test environment, you've probably come across [github.com/ory/dockertest](https://github.com/ory/dockertest) package. There's also [testcontainers](https://golang.testcontainers.org) for setting up containers. Alternatively, you could even invoke docker as a command and use that. Whichever your poison, the approach will look similar. We'll use *dockertest* for our examples. + +Usually, the first thing you do is set up something to act as the client. With *dockertest* it means creating a *dockertest.Pool*. And we need to set it up in our *TestMain*: + +```go +var dockerPool *dockertest.Pool + +func TestMain(m *testing.M) { + var err error + pool, err = dockertest.NewPool("") + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + + // Set a time for our retries. A lower value probably makes more sense. + pool.MaxWait = 120 * time.Second + code := m.Run() + os.Exit(code) +} +``` + +If we are writing tests, then using a specific helper is going to be very convenient. + +```go +func TestCreateTable(t *testing.T) { + ctx := context.Background() + WithDatabase(ctx, t, func(t *testing.TB, db *pgx.Conn) { + _, err := db.Exec(ctx, ` + CREATE TABLE accounts ( user_id serial PRIMARY KEY ); + `) + if err != nil { + t.Fatal(err) + } + }) +} + +func WithDatabase[TB testing.TB](ctx context.Context, tb TB, test func(t TB, db *pgx.Conn)) { + // < snip > +} +``` + +This approach creates a docker image and calls *test* callback whenever it's ready. + +The callback based approach is especially helpful if you need to test with multiple backends such as Cockroach and Postgres. In your own codebase you probably would return the data layer interface rather than *\*pgx.Conn* directly. For example: + +```go +func TestCreateTable(t *testing.T) { + ctx := context.Background() + db := NewDatabase(ctx, t) + _, err := db.Exec(ctx, ` + CREATE TABLE accounts ( user_id serial PRIMARY KEY ); + `) + if err != nil { + t.Fatal(err) + } +} + +func NewDatabase(ctx context.Context, tb testing.TB) *pgx.Conn { + // create the database resource + tb.Cleanup(func() { + err := db.Close(ctx) + if err != nil { + tb.Logf("failed to close db: %v", err) + } + }) + return conn +} +``` + +A single table migration isn't indicative of a proper database layer, but it's sufficient for seeing the best-case scenario. Adding more tables didn't seem to affect things that much. + +Let's get back on track and see how you can implement the first approach. It's should be trivial to convert one to the other: + +```go +func WithDatabase[TB testing.TB](ctx context.Context, tb TB, test func(t TB, db *pgx.Conn)) { + // First we need to specify the image we wish to use. + resource, err := dockerPool.RunWithOptions(&dockertest.RunOptions{ + Repository: "postgres", + Tag: "15", + Env: []string{ + "POSTGRES_PASSWORD=secret", + "POSTGRES_USER=user", + "POSTGRES_DB=main", + "listen_addresses = '*'", + }, + }, func(config *docker.HostConfig) { + // set AutoRemove to true so that stopped container goes away by itself + config.AutoRemove = true + config.RestartPolicy = docker.RestartPolicy{Name: "no"} + }) + if err != nil { + tb.Fatalf("Could not start resource: %s", err) + } + defer func() { + if err := dockerPool.Purge(resource); err != nil { + tb.Logf("failed to stop: %v", err) + } + }() + + // Construct our connection string. + hostAndPort := resource.GetHostPort("5432/tcp") + databaseConnstr := fmt.Sprintf("postgres://user:secret@%s/main?sslmode=disable", hostAndPort) + + err = resource.Expire(2 * 60) // hard kill the container after 2 minutes, just in case. + if err != nil { + tb.Fatalf("Unable to set container expiration: %v", err) + } + + // Finally, try to connect to the container. + // We need to retry, because it might take some time until the container becomes available. + var db *pgx.Conn + err = dockerPool.Retry(func() error { + db, err = pgx.Connect(ctx, databaseConnstr) + if err != nil { + return err + } + return nil + }) + if err != nil { + tb.Fatal("unable to connect to Postgres", err) + } + + defer func() { + err := db.Close(ctx) + if err != nil { + tb.Logf("failed to close db: %v", err) + } + }() + + // Finally call our test code. + test(tb, db) +} +``` + +Let's look at the performance: + +``` +| Environment | Test | Time | +| ---------------------------- | ---------- | ------------ | +| Windows Threadripper 2950X | Container | 2.86s ± 6% | +| MacOS M1 Pro | Container | 1.63s ± 16% | +| Linux Xeon Gold 6226R | Container | 2.24s ± 10% | +``` + +## Using DATABASE + +In most cases, creating a new postgres instance per test isn't necessary. It'll be entirely sufficient to have a database per test. If we have SUPERUSER permissions in postgres we can create them dynamically. + +To contrast with the previous approach, let's use a locally installed Postgres instance. This can be helpful, if you want to run tests against a remote database or want to avoid the container startup time. + +```go +var pgaddr = flag.String("database", os.Getenv("DATABASE_URL"), "database address") +``` + +Let's rewrite the function to create a new database per test: + +```go +func WithDatabase[TB testing.TB](ctx context.Context, tb TB, test func(t TB, db *pgx.Conn)) { + if *pgaddr == "" { + tb.Skip("-database flag not defined") + } + dbaddr := *pgaddr + + // We need to create a unique database name so that our parallel tests don't clash. + var id [8]byte + rand.Read(id[:]) + uniqueName := tb.Name() + "/" + hex.EncodeToString(id[:]) + + // Create the main connection that we use to create the database. + maindb, err := pgx.Connect(ctx, dbaddr) + if err != nil { + tb.Fatalf("Unable to connect to database: %v", err) + } + + // Run the database creation query and defer the database cleanup query. + if err := createDatabase(ctx, maindb, uniqueName); err != nil { + tb.Fatalf("unable to create database: %v", err) + } + defer func() { + if err := dropDatabase(ctx, maindb, uniqueName); err != nil { + tb.Fatalf("unable to drop database: %v", err) + } + }() + + // Modify the connection string to use a different database. + connstr, err := connstrWithDatabase(dbaddr, uniqueName) + if err != nil { + tb.Fatal(err) + } + + // Create a new connection to the database. + db, err := pgx.Connect(ctx, connstr) + if err != nil { + tb.Fatalf("Unable to connect to database: %v", err) + } + defer func() { _ = db.Close(ctx) }() + + // Run our test code. + test(tb, db) +} +``` + +Now for the small utility funcs that we used: + +```go +// connstrWithDatabase changes the main database in the connection string. +func connstrWithDatabase(connstr, database string) (string, error) { + u, err := url.Parse(connstr) + if err != nil { + return "", fmt.Errorf("invalid connstr: %q", connstr) + } + u.Path = database + return u.String(), nil +} + +// createDatabase creates a new database with the specified name. +func createDatabase(ctx context.Context, db *pgx.Conn, name string) error { + _, err := db.Exec(ctx, `CREATE DATABASE `+sanitizeDatabaseName(name)+`;`) + return err +} + +// dropDatabase drops the specific database. +func dropDatabase(ctx context.Context, db *pgx.Conn, name string) error { + _, err := db.Exec(ctx, `DROP DATABASE `+sanitizeDatabaseName(name)+`;`) + return err +} + + +// sanitizeDatabaseName is ensures that the database name is a valid postgres identifier. +func sanitizeDatabaseName(schema string) string { + return pgx.Identifier{schema}.Sanitize() +} +``` + +The performance looks already significantly better: + +| Environment | Test | Time | +| ---------------------------- | ---------- | ------------ | +| Windows Threadripper 2950X | Container | 2.86s ± 6% | +| Windows Threadripper 2950X | Database | 136ms ± 12% | +| MacOS M1 Pro | Container | 1.63s ± 16% | +| MacOS M1 Pro | Database | 136ms ± 12% | +| Linux Xeon Gold 6226R | Container | 2.24s ± 10% | +| Linux Xeon Gold 6226R | Database | 135ms ± 10% | + +## Using SCHEMA + +But, 90ms is still a lot of time per single test. There's one lesser-known approach we discovered in Storj. It's possible to use a [schema](https://www.postgresql.org/docs/current/ddl-schemas.html) to create an isolated namespace that can be dropped together. + +Creating a new schema is as straightforward as executing `CREATE SCHEMA example;` and dropping `DROP SCHEMA example CASCADE;`. When connecting to the database it's possible to add a connection string parameter `?search\_path=example` to execute all queries by default in that schema. + +Of course, if you use schemas for other purposes in your system, then this approach may complicate the rest of your code. Similarly, schemas are not as isolated as separate databases. + +Now that the disclaimer is out of the way, let's take a look at some code: + +```go +func WithSchema[TB testing.TB](ctx context.Context, tb TB, test func(t TB, db *pgx.Conn)) { + if *pgaddr == "" { + tb.Skip("-database flag not defined") + } + dbaddr := *pgaddr + + // We need to create a unique schema name so that our parallel tests don't clash. + var id [8]byte + rand.Read(id[:]) + uniqueName := tb.Name() + "/" + hex.EncodeToString(id[:]) + + // Change the connection string to use a specific schema name. + connstr, err := connstrWithSchema(dbaddr, uniqueName) + if err != nil { + tb.Fatal(err) + } + db, err := pgx.Connect(ctx, connstr) + if err != nil { + tb.Fatalf("Unable to connect to database: %v", err) + } + defer func() { _ = db.Close(ctx) }() + + // Surprisingly, it's perfectly fine to create a schema after connecting with the name. + if err := createSchema(ctx, db, uniqueName); err != nil { + tb.Fatal(err) + } + defer func() { + if err := dropSchema(ctx, db, uniqueName); err != nil { + tb.Fatal(err) + } + }() + + test(tb, db) +} +``` + +The smaller utilities that make it work: + +```go +// connstrWithSchema adds search_path argument to the connection string. +func connstrWithSchema(connstr, schema string) (string, error) { + u, err := url.Parse(connstr) + if err != nil { + return "", fmt.Errorf("invalid connstr: %q", connstr) + } + u.Query().Set("search_path", sanitizeSchemaName(schema)) + return u.String(), nil +} + +// createSchema creates a new schema in the database. +func createSchema(ctx context.Context, db *pgx.Conn, schema string) error { + _, err := db.Exec(ctx, `CREATE SCHEMA IF NOT EXISTS`+sanitizeSchemaName(schema)+`;`) + return err +} + +// dropSchema drops the specified schema and associated data. +func dropSchema(ctx context.Context, db *pgx.Conn, schema string) error { + _, err := db.Exec(ctx, `DROP SCHEMA `+sanitizeSchemaName(schema)+` CASCADE;`) + return err +} + +// sanitizeSchemaName is ensures that the name is a valid postgres identifier. +func sanitizeSchemaName(schema string) string { + return pgx.Identifier{schema}.Sanitize() +} +``` + +After running some benchmarks we can see that we've reached ~20ms: + +| Environment | Test | Time | +| ---------------------------- | ---------- | ------------- | +| Windows Threadripper 2950X | Container | 2.86s ± 6% | +| Windows Threadripper 2950X | Database | 136ms ± 12% | +| Windows Threadripper 2950X | Schema | 26.7ms ± 3% | +| MacOS M1 Pro | Container | 1.63s ± 16% | +| MacOS M1 Pro | Database | 136ms ± 12% | +| MacOS M1 Pro | Schema | 19.7ms ± 20% | +| Linux Xeon Gold 6226R | Container | 2.24s ± 10% | +| Linux Xeon Gold 6226R | Database | 135ms ± 10% | +| Linux Xeon Gold 6226R | Schema | 29.2ms ± 16% | + + +## Final tweaks + +There's one important flag that you can adjust in Postgres to make it run faster... of course, this should only be used for testing. It's disabling [fsync](https://www.postgresql.org/docs/current/runtime-config-wal.html). + +The final results of the comparison look like: + +| Environment | Test | fsync | Time | +| ---------------------------- | ---------- | ------ | --------------- | +| Windows Threadripper 2950X | Container | on | 2.86s ± 6% | +| Windows Threadripper 2950X | Container | off | 2.82s ± 4% | +| Windows Threadripper 2950X | Database | on | 136ms ± 12% | +| Windows Threadripper 2950X | Database | off | 105ms ± 30% | +| Windows Threadripper 2950X | Schema | on | 26.7ms ± 3% | +| Windows Threadripper 2950X | Schema | off | 20.5ms ± 5% | +| MacOS M1 Pro | Container | on | 1.63s ± 16% | +| MacOS M1 Pro | Container | off | 1.64s ± 13% | +| MacOS M1 Pro | Database | on | 136ms ± 12% | +| MacOS M1 Pro | Database | off | 105ms ± 30% | +| MacOS M1 Pro | Schema | on | 19.7ms ± 20% | +| MacOS M1 Pro | Schema | off | 18.5ms ± 31% | +| Linux Xeon Gold 6226R | Container | on | 2.24s ± 10% | +| Linux Xeon Gold 6226R | Container | off | 1.97s ± 10% | +| Linux Xeon Gold 6226R | Database | on | 135ms ± 10% | +| Linux Xeon Gold 6226R | Database | off | 74.2ms ± 10% | +| Linux Xeon Gold 6226R | Schema | on | 29.2ms ± 16% | +| Linux Xeon Gold 6226R | Schema | off | 15.3ms ± 15% | + +All the tests were run in a container that didn't have persistent disk mounted. The fsync=off would probably have a bigger impact with an actual disk. + +So for the conclusion, we looked at three different approaches to creating a clean Postgres environment. The approaches aren't completely equivalent, but use the fastest one that you can. + + + diff --git a/app/(blog)/blog/introducing-drpc-our-replacement-for-grpc/26c69fe77df6e712.png b/app/(blog)/blog/introducing-drpc-our-replacement-for-grpc/26c69fe77df6e712.png new file mode 100644 index 000000000..28b240d00 Binary files /dev/null and b/app/(blog)/blog/introducing-drpc-our-replacement-for-grpc/26c69fe77df6e712.png differ diff --git a/app/(blog)/blog/introducing-drpc-our-replacement-for-grpc/e2c929baac38fe20.png b/app/(blog)/blog/introducing-drpc-our-replacement-for-grpc/e2c929baac38fe20.png new file mode 100644 index 000000000..d9403a45f Binary files /dev/null and b/app/(blog)/blog/introducing-drpc-our-replacement-for-grpc/e2c929baac38fe20.png differ diff --git a/app/(blog)/blog/introducing-drpc-our-replacement-for-grpc/page.md b/app/(blog)/blog/introducing-drpc-our-replacement-for-grpc/page.md new file mode 100644 index 000000000..dfdfb5f46 --- /dev/null +++ b/app/(blog)/blog/introducing-drpc-our-replacement-for-grpc/page.md @@ -0,0 +1,148 @@ +--- +author: + name: JT Olio and Jeff Wending +date: '2021-04-27 00:00:00' +heroimage: ./e2c929baac38fe20.png +layout: blog +metadata: + description: In 2016, Google launched gRPC, which has overall taken the systems + programming community by storm. gRPC stands for something with a G, Remote Procedure + Call; it's a mechanism for easily defining interfaces between two different remote + services. Building a new decentralized storage platform from the ground up in + Go, obviously, we considered using gRPC to simplify our development process in + peer-to-peer remote procedure calling. In fact, I'm not even sure we really considered + anything else. Fast forward to the latter half of 2019, and we had 170k lines + of Go, a beta network of over 4 PB, real live active users, and it turns out the + gRPC bed we made for ourselves was not all roses. So we rewrote gRPC and migrated + our live network. DRPC is an open-source, drop-in replacement that handles everything + we needed from gRPC (and most likely, everything you need) in under 3000 lines + of Go. It now powers our full network of tens of thousands of servers and countless + clients. + title: 'Introducing DRPC: Our Replacement for gRPC' +title: 'Introducing DRPC: Our Replacement for gRPC' + +--- + +In 2016, Google launched [gRPC](https://grpc.io/), which has overall taken the systems programming community by storm. gRPC stands for something with a G, Remote Procedure Call; it's a mechanism for easily defining interfaces between two different remote services. It's tightly bundled with [Protocol Buffers](https://developers.google.com/protocol-buffers) version 3 (another highly adopted data interchange specification from Google), and... it seems like everyone is using it. Wikipedia, Square, Netflix, IBM, Docker, Cockroach Labs, Cisco, Spotify, Dropbox, etc., all use gRPC. + + +Here at Storj, we’re pioneers in decentralized cloud storage. By early 2018, we built and scaled a 150 petabyte decentralized storage network. Of course, like every good scaling story, by the time we got to 150 petabytes, we discovered some fundamental architectural issues that needed to be reworked. Staring down the barrel of a few hundred thousand lines of untyped Javascript with... sort of decent test coverage, we made the [risky decision](https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/) in March of 2018 to fix these architectural issues with a ground-up reimplementation in Go. We're calling this iteration our V3 network, which had its production launch in March of 2020. You can read all about our architecture in [our whitepaper](https://storj.io/storjv3.pdf), and you can [try out our live service](https://www.storj.io/). + + +*(Aside:* [*We’re hiring engineers!*](https://storj.io/careers/)*)* + +Building a new decentralized storage platform from the ground up in Go, obviously, we considered using gRPC to simplify our development process in peer-to-peer remote procedure calling. In fact, I'm not even sure we really considered anything else. Using gRPC for us was a deliberate decision to [avoid using an innovation token](https://mcfunley.com/choose-boring-technology). How could gRPC be the wrong choice? It has impressive credentials and wide usage. We were always disappointed Google didn't release a standard RPC implementation with proto2. With an otherwise previously strongly positive experience with protocol buffers, we were excited to jump all in to the new protobuf RPC land. + +Fast forward to the latter half of 2019, and we had 170k lines of Go, a beta network of over 4 PB, real live active users, and it turns out the gRPC bed we made for ourselves was not all roses, and we kind of pooped just a little in it. Just a little bit. This much ->||<-. So not a lot, but still. + +So we rewrote gRPC and migrated our live network. [DRPC](https://storj.github.io/drpc/) is an open-source, drop-in replacement that handles everything we needed from gRPC (and most likely, everything you need) in under 3000 lines of Go. It now powers our full network of tens of thousands of servers and countless clients. + + +[Check out DRPC here!](https://storj.github.io/drpc/) + +# Where gRPC needs improvement + +Let’s just get out here and say what not enough people are saying—in a nutshell, gRPC has feature creep, bloat and is trying to solve too many problems. It’s overcomplicated and has become a dumping ground of features, tangled in a standards body-based web of crap.  + +Let’s go over the problems with gRPC one by one! + + +## Feature bloat + +Did you know that there are 40 different dial options? There are 26 server options. You have 13 call options. gRPC is huge. One major issue that led us away from gRPC was that it constituted over a fifth of our own (sizeable) binaries. + +Do you use gRPC’s built-in APIs for internal load balancing? You probably don’t—your load balancer probably does something else. Do you use manual name resolution builders? Do you find yourself uncertain what operations are synchronous or not? Wait, should we use WithBlock? gRPC tends to accrue features so that it tries to solve every problem, and with every solution comes more code to maintain, places for bugs to hide, and additional semantics to worry about. + +On the other hand, DRPC’s core is under 3000 lines! It’s a reasonable task to audit and understand it. + + +## Deprecation + + + + +![](./26c69fe77df6e712.png) + +This tweet is from 2019, and as of today, in 2021, WithDefaultServiceConfig is still experimental, and WithBalancerName is still deprecated. + +At the time of this writing, there are 37 deprecation notices in the top-level [gRPC documentation](https://pkg.go.dev/google.golang.org/grpc). This makes it hard to understand what you’re supposed to use, what you’re not supposed to use, what things are available, etc. + + +## High resource usage + +81% of the heap usage of one of our Storage Nodes was in gRPC. You can’t use another protobuf library; there are a large number of allocations, you get its own HTTP/2 server, and the hits keep coming.  + + +A protobuf-based protocol has the fortunate ability to avoid a complicated string parsing and overhead of traditional, older protocols, such as HTTP, unless you use gRPC. Then you have to deal with HTTP/2 and all the legacy edge cases that may arise.  + + +## Opacity + +Pop quiz: imagine you’re trying to debug some issue that’s happening during dialing of connections, and you use . See if you can find where in the gRPC code base the function you provide there is called. + + +gRPC is at least 10 times more lines of code than DRPC, and it is safe to say some of the API has grown organically and is hard to reason about. + + +DRPC, meanwhile, is implemented to be lightweight, straightforward, clear, and easy to debug. + + +# Wait, but a rewrite? + +Yep! Considering we have 170k lines of Go, tightly integrated into both single and streaming request styles of gRPC, in 2019, we narrowed our options down to: + + +* [Thrift](https://thrift.apache.org/) +* [Twirp](https://github.com/twitchtv/twirp) +* Fork gRPC? +* Write our own + +We really wanted to avoid having to change every service already registered with gRPC. Again, the services were fine, and we just needed to change the connection layer. Thrift was a pretty big departure for pretty much all of our code, so we eliminated it. Maybe it would have been good to start with, but we judged by the cover and suspected it wasn’t the best place to start. + + +We could have eliminated Twirp for the same reason, but Twirp had another problem - we needed support for bidirectional streaming, and Twirp didn’t have it. + + +Forking gRPC may have been a good choice, but we would suddenly be responsible for all of it, as we ripped out the parts we didn’t need. Ripping out the overhead of HTTP/2 alone was by itself essentially a rewrite. It seemed like a simpler undertaking to start fresh. + + +So, we decided to time-box an experiment to write our own. The experiment was a smashing success. + +# DRPC + +[DRPC](https://storj.github.io/drpc/) is a code-wise drop-in replacement for the client/server interactions of gRPC. If you’re using gRPC today in Go, you should be able to swap your protocol buffer generation pipeline to DRPC and be on your way. If you already have proto3 .proto files, the protoc protobuf compiler can be told to generate DRPC code instead of gRPC code (or both, if you're migrating). + +DRPC supports a wide range of functionality in its spartan few thousand lines. DRPC is blazingly fast and lightweight (the protocol does not require HTTP header parsing, for example), it supports unitary and streaming requests, it has an HTTP/JSON gateway, it supports metadata for per-request side-channel information like tracing, it supports layering and middleware, etc. + +[Check out how easy our Quickstart documentation is!](https://storj.github.io/drpc/docs.html) + +Also be sure to check out the gRPC vs DRPC benchmarks on [our Github README](https://github.com/storj/drpc#readme). I want to specifically call out how much better with memory usage DRPC is. GC pressure sucks! When you have high performance servers, reducing GC pressure is always a good call. + +Also make sure to see more examples at   + +It's worth pointing out that DRPC is *not* the same protocol as gRPC, and DRPC clients cannot speak to gRPC servers and vice versa. + +# Migration + +One major challenge we faced was that we already had gRPC deployed. We needed to support both DRPC and gRPC clients for a transition period until everything understood DRPC. + + +As a result, we wrote (and included in DRPC) migration helpers that allow you to listen for and respond to DRPC and gRPC requests on the same port. Make sure to check out and our gRPC and DRPC example: + + +Here were our transition steps: + + +1. Release and deploy new server code that understands both gRPC and DRPC concurrently. With DRPC, this was a breeze since all of our application code could be used identically with both, and our ListenMux allowed us to do both from the same server port. +2. Once all the servers and Nodes were updated, release and deploy new clients that spoke DRPC instead of gRPC. +3. Once all of the old clients were gone, we removed the gRPC code. + +We immediately eliminated a whole class of inscrutable WAN network errors (our availability metrics went up), improved performance and reduced CPU and resource utilization, reduced our binary sizes significantly, and have overall been much, much happier. + +# We're open source + +DRPC, like almost everything else at Storj, is open source. DRPC is MIT/expat licensed, and we’d love your help! Since we currently only have Go bindings for DRPC, bindings for new languages would be a great place to start. + +Feel free to check out [our Github repo](https://github.com/storj/drpc) and let us know if we can help you dive in! + + diff --git a/app/(blog)/blog/lensm/color-selection.png b/app/(blog)/blog/lensm/color-selection.png new file mode 100644 index 000000000..1501b36e7 Binary files /dev/null and b/app/(blog)/blog/lensm/color-selection.png differ diff --git a/app/(blog)/blog/lensm/compiler-explorer.png b/app/(blog)/blog/lensm/compiler-explorer.png new file mode 100644 index 000000000..23d1ad877 Binary files /dev/null and b/app/(blog)/blog/lensm/compiler-explorer.png differ diff --git a/app/(blog)/blog/lensm/expose-objfile.png b/app/(blog)/blog/lensm/expose-objfile.png new file mode 100644 index 000000000..d9881849e Binary files /dev/null and b/app/(blog)/blog/lensm/expose-objfile.png differ diff --git a/app/(blog)/blog/lensm/final-result.png b/app/(blog)/blog/lensm/final-result.png new file mode 100644 index 000000000..9c000a91b Binary files /dev/null and b/app/(blog)/blog/lensm/final-result.png differ diff --git a/app/(blog)/blog/lensm/first-output.png b/app/(blog)/blog/lensm/first-output.png new file mode 100644 index 000000000..40acf9f9d Binary files /dev/null and b/app/(blog)/blog/lensm/first-output.png differ diff --git a/app/(blog)/blog/lensm/hero.jpeg b/app/(blog)/blog/lensm/hero.jpeg new file mode 100644 index 000000000..a5184b0a4 Binary files /dev/null and b/app/(blog)/blog/lensm/hero.jpeg differ diff --git a/app/(blog)/blog/lensm/internal-objfile.png b/app/(blog)/blog/lensm/internal-objfile.png new file mode 100644 index 000000000..ab4397bc2 Binary files /dev/null and b/app/(blog)/blog/lensm/internal-objfile.png differ diff --git a/app/(blog)/blog/lensm/jump-1.png b/app/(blog)/blog/lensm/jump-1.png new file mode 100644 index 000000000..d1c2f3fe3 Binary files /dev/null and b/app/(blog)/blog/lensm/jump-1.png differ diff --git a/app/(blog)/blog/lensm/jump-2.png b/app/(blog)/blog/lensm/jump-2.png new file mode 100644 index 000000000..a0877387a Binary files /dev/null and b/app/(blog)/blog/lensm/jump-2.png differ diff --git a/app/(blog)/blog/lensm/jump-3.png b/app/(blog)/blog/lensm/jump-3.png new file mode 100644 index 000000000..0f3c10b4b Binary files /dev/null and b/app/(blog)/blog/lensm/jump-3.png differ diff --git a/app/(blog)/blog/lensm/jump-4.png b/app/(blog)/blog/lensm/jump-4.png new file mode 100644 index 000000000..ceeeddabb Binary files /dev/null and b/app/(blog)/blog/lensm/jump-4.png differ diff --git a/app/(blog)/blog/lensm/meld.png b/app/(blog)/blog/lensm/meld.png new file mode 100644 index 000000000..6dd5354fd Binary files /dev/null and b/app/(blog)/blog/lensm/meld.png differ diff --git a/app/(blog)/blog/lensm/objdump-output.png b/app/(blog)/blog/lensm/objdump-output.png new file mode 100644 index 000000000..9e14ee676 Binary files /dev/null and b/app/(blog)/blog/lensm/objdump-output.png differ diff --git a/app/(blog)/blog/lensm/page.md b/app/(blog)/blog/lensm/page.md new file mode 100644 index 000000000..dfd4271ac --- /dev/null +++ b/app/(blog)/blog/lensm/page.md @@ -0,0 +1,148 @@ +--- +author: + name: Egon Elbre +date: '2022-07-18 00:00:00' +heroimage: ./hero.jpeg +layout: blog +metadata: + description: "I couldn\u2019t find a great tool for viewing disassembly, so I wrote\ + \ it myself over the weekend." + title: Lensm, A Tool for Viewing Disassembly +title: Lensm, A Tool for Viewing Disassembly + +--- + +I couldn’t find a great tool for viewing disassembly, so I [wrote it myself over the weekend](https://github.com/loov/lensm). + +At Storj, we are constantly looking for ways to accelerate our team’s efficiency, and one of those is building the tools we need. + +One of the major things you will rub against when you delve into performance optimization is viewing the assembly that the compiler generates. It's usually not efficient to write assembly yourself, and it's better to try to coerce the compiler to produce the assembly you want. Here's my story of writing a little tool for viewing disassembly. + +![](./screenshot.gif) + +# Getting Annoyed + +My story starts on a weekend when I was doing a bunch of tiny optimizations to the [Gio UI](https://gioui.org/) project. There are ways to view the assembly; one is to use **go tool objdump -s funcname** from the command line. However, it's rather difficult to see how the source code and assembly are related. + +![](./objdump-output.png) + +There is an excellent online tool for writing code and seeing the output [https://go.godbolt.org](https://go.godbolt.org/). The visuals are much clearer. + +The corresponding lines of code have the same color. When you hover over the specific lines of code, the corresponding assembly is also highlighted. + +![](./compiler-explorer.png) + +Compiler Explorer has many other nice features as well: sharing the result, compiling with different versions, diffing output from different compilers, and description of assembly instructions. The amount of different languages and compilers is staggering. + +Despite how nice Compiler Explorer is, it's still an online tool, and you need to copy-paste your relevant code to the explorer. + +After trying many times, my annoyance finally kicked in: + +*"Someone should've written this tool already–it shouldn't be too difficult."* + +Over the years of developing, I've found that getting annoyed is a rather excellent way to start a new project. + +# Disassembly + +The first step in the project was to have access to the disassembly. It would be wasteful to start a disassembler from scratch. I knew that **go tool objdump** could already do it, so maybe they have some library they are using. + +![](./internal-objfile.png) + +Indeed, they are using a library, but it's internal to the compiler. The internal library looks pretty nice to use as well. I guess I need to extract it for my own needs. Copying the relevant code and adjusting the import paths was grunt work, but I got it extracted. Luckily the license for the Go code is open-source. + +I needed to expose a little bit more information from the API to access the [necessary details](https://github.com/loov/lensm/commit/5bb596225accd3d6c0b4dbc13c4e6189c558c879#diff-1596bd8ceb74246828aacab827b39a33075c86baa627fbbeb7491bd31eef1169), but I got it working. Here's the debug print from the initial output: + +![](./expose-objfile.png) + +Of course, extracting the internals means needing to keep it manually updated. I'm sure there was a tool to rewrite the paths and keep them automatically updated. Alternatively, maybe the Go project would accept a patch that exposes the information in some JSON format so the visualizer can call the appropriate Go compiler. But all of that is a project for another day. + +## Extracting Source Code + +The first important step was to figure out the relevant source code that needed to be loaded. This seems a relatively easy thing in concept. It's mainly "Collect the list of lines per source file". However, the gotcha is how to represent the data, and similarly, you probably don't want just the lines but also some of the surrounding code. + +This is the basic structure for representing source: + +![](./source-definition.png) + +Every assembly function can have multiple associated **Source** files due to inlining. Similarly, the code needed from different files isn't contiguous, and you wouldn't want to show more than is required. + +Most of the data munging is: collect all the source lines, convert them into ranges, expand the ranges (for the surrounding context). We also need to do it in reverse: figure out which lines in disassembly correspond to the source code. Note that each source line can correspond to multiple disassembly lines, and they might not be contiguous. + +Once I got it working, I did a debug print of the relevant source lines: + +![](./source-definition-output.png) + +# Drawing Code + +I was trying to optimize the code for [Gio UI](https://gioui.org/), so of course, it was a natural choice for building a tool such as this. It has pretty lovely drawing capabilities that I'll need. + +The question was then, how should it be visualized. Compiler Explorer visualization is a great starting point. However, it's not as clear as I would like it to be. When starting the project, I already had a design in mind. There are many source diffing tools that offer visualizing related lines. For example, here is what Meld tool looks like: + +![](./meld.png) + +There are other tools such as Kompare, CodeCompare, Oxygen Compare that offer similar visualization. I really like how it shows how one side is related to the other. To draw the shape, we can use the following idea: + +![](./relation-shape.png) + +*The purple lines show the final relation shape. The orange arrows show bezier curve handles.* + +Drawing the visuals seemed then straightforward: + +1. figure out the location of each line of the source and assembly; +2. draw the relation shape for each line of source and related assembly lines; +3. draw the text on top of the relation shapes. + +One difficult thing people encounter with such projects is: how to choose a random color such that they are distinct, visually pleasing, and code is easy to write. One nice trick I've picked up over time is this formula: + +*hue: index \* phi \* 2 \* PI, saturation: 60%, lightness: 60%* + +You can adjust the saturation and lightness between 50% to 90% to get different lightness and saturation. If you want a more pastel color look, you would use a lower saturation and higher lightness. For dark mode, you would use lightness below 30%. (The color selection assumes that hue is defined with the range 0 .. 2\*PI). There are a few variations of the hue selection: + +![](./color-selection.png) + +As you can see, the 𝜑 = 1.618033988749… constant allows selecting values on a hue circle such that sequential numbers are different and won't repeat. If you want a smoother transition, then using i × 1/𝜑 works a treat. If you want more contrast, then i × 𝜑 × 2𝜋 is nicer. + +Once you put all these ideas together, you get the first output: + +![](./first-output.png) + +I also added a small interaction – when you hover the mouse over a line of code, it highlights the relation shape. + +## Drawing Jumps + +The next thing I wanted to visualize was drawing jumps in the code. They are important from a performance perspective. It's relatively common for disassemblers to draw an arrow from the jump location to the destination. This brings up two problems, detecting the jumps, and figuring out how to draw the lines. + +Unfortunately, the objfile library disassembler doesn't expose the information whether the instruction is a jump and when it jumps, then where to. I didn't want to dig too deep into this, so I reached for the usual tool for this – regular expression matching. It seemed that all the jumps ended with a hex number, such as **JMP 0x123**... of course, that approach broke. On arm processors, they look like **BLS 38(PC)**. I added a special case for it for now, but it'll probably break again on some other platform. + +To draw the jumps initially, I just drew them like a stack. In other words, push the jump line to the sidebar when you encounter one and then pop it when it ends. Of course, that didn't look great due to overlapping lines: + +![](./jump-1.png) + +In some cases it even caused the lines to be on top of each other. I searched for a nice algorithm for drawing them; however, I came up empty. Finally, I decided to go with the following approach, sort all the jump ranges based on their starting and ending point. If multiple ranges start from the same location, the larger range is sorted first. Then divide the sidebar into lanes; every new range picks the first lane that is free – starting from the left. This ends up minimizing crossings. + +![](./jump-2.png) + +It's by no means ideal. It can still draw the jump line too far from the code. + +![](./jump-3.png) + +Or do this thing here: + +![](./jump-4.png) + +But, these are things someone will fix some other day. + +# Summary + +After a few days of work, I have a nice tool for viewing disassembly. + +![](./final-result.png) + +Choosing a name was also a struggle. I wanted it to be easily recognizable and searchable. I asked in Gophers #performance channel and Jan Mercl suggested "lensm," which is "lens" and "asm" smushed together. + +When you look at the code and think: "For a performance-conscious project, it doesn't look very efficient – allocations and suboptimal algorithms everywhere. Also, the code looks very messy." + +That's all true, but the goal was to get it done quickly. And, if I do need to optimize, then I have an extra tool in my toolbelt to optimize it. + +I'll still have a few things I want to add before I can call it sufficiently complete. Nevertheless, it's already functional, so give it a test run at . If you feel like something is missing, then come along for the ride and submit a patch; there have already been a few contributors. + diff --git a/app/(blog)/blog/lensm/relation-shape.png b/app/(blog)/blog/lensm/relation-shape.png new file mode 100644 index 000000000..dc213a25a Binary files /dev/null and b/app/(blog)/blog/lensm/relation-shape.png differ diff --git a/app/(blog)/blog/lensm/screenshot.gif b/app/(blog)/blog/lensm/screenshot.gif new file mode 100644 index 000000000..7053e4289 Binary files /dev/null and b/app/(blog)/blog/lensm/screenshot.gif differ diff --git a/app/(blog)/blog/lensm/source-definition-output.png b/app/(blog)/blog/lensm/source-definition-output.png new file mode 100644 index 000000000..04585c5b5 Binary files /dev/null and b/app/(blog)/blog/lensm/source-definition-output.png differ diff --git a/app/(blog)/blog/lensm/source-definition.png b/app/(blog)/blog/lensm/source-definition.png new file mode 100644 index 000000000..06e4deb89 Binary files /dev/null and b/app/(blog)/blog/lensm/source-definition.png differ diff --git a/app/(blog)/blog/open-source-and-open-data-storj-dcs-network-statistics/20487c1035ee937f.jpeg b/app/(blog)/blog/open-source-and-open-data-storj-dcs-network-statistics/20487c1035ee937f.jpeg new file mode 100644 index 000000000..464dd99e0 Binary files /dev/null and b/app/(blog)/blog/open-source-and-open-data-storj-dcs-network-statistics/20487c1035ee937f.jpeg differ diff --git a/app/(blog)/blog/open-source-and-open-data-storj-dcs-network-statistics/page.md b/app/(blog)/blog/open-source-and-open-data-storj-dcs-network-statistics/page.md new file mode 100644 index 000000000..32491d1b2 --- /dev/null +++ b/app/(blog)/blog/open-source-and-open-data-storj-dcs-network-statistics/page.md @@ -0,0 +1,84 @@ +--- +author: + name: Brandon Iglesias +date: '2021-08-24 00:00:00' +heroimage: ./20487c1035ee937f.jpeg +layout: blog +metadata: + description: "We recently began publicly exposing more data about the network in\ + \ a way that could be used on-demand and programmatically. If you missed it, we\ + \ have started publishing what we think is the most important network statistics\ + \ on our new Storj DCS Public Network Statistics page. Now, if you\u2019re a non-technical\ + \ person, this may not be what you expected. Here\u2019s an explanation of why\ + \ we took this approach." + title: 'Open Source and Open Data: Storj DCS Network Statistics' +title: 'Open Source and Open Data: Storj DCS Network Statistics' + +--- + +You might often see or hear us reference our company values. The fact of the matter is that our values—including openness, transparency, and empowering our community—are what drives us as a company and as individuals. Our values are our north star, so when faced with decisions or when we find ourselves at a crossroads, we often reexamine the situation through the lens of our company values.  + + +Our company value of Open means we’re committed to the free and open sharing of software, information, knowledge, and ideas. It’s been shown this kind of openness yields better results in the long run—not just for the company but for the industry and community as well. Open source software has been the cornerstone for innovations such as containers and microservices, private web browsing, and new databases that enable other powerful services. This is why [we are committed to open source software](https://www.storj.io/open-source).   + + +Since the launch of Storj DCS, our community has been asking for more statistics and data on the network. Some folks in our community have even found ways of reverse-engineering the network to derive statistics about it. A great example of this ingenuity is [Storj Net Info](https://storjnet.info/). Providing these statistics has always been a goal of ours, but the task has been lower on our priority roadmap list than delivering some other critical features that Storj DCS customers need.  + + +We [recently](https://forum.storj.io/t/publicly-exposed-network-data-official-statistics-from-storj-dcs-satellites/14103) began publicly exposing more data about the network in a way that could be used on-demand and programmatically. If you missed it, we have started publishing what we think is the most important network statistics on our new [Storj DCS Public Network Statistics](https://stats.storjshare.io/) page. Now, if you’re a non-technical person, this may not be what you expected. Here’s an explanation of why we took this approach.  + + +New members of our community often ask why don't we build a service like Dropbox or Google Drive instead of a cloud object storage service like Storj DCS. This is because we’re focused on providing the building blocks (underlying storage layer) for others to build those kinds of applications. By doing this, we can enable dozens of companies to build Dropbox-like services on Storj DCS (or easily migrate their existing applications to the service). + +We decided we wanted to take a similar approach with these statistics, so we’re exposing the data in JSON format instead of just providing a dashboard for people to view. On this page, you’ll find statistics such as the amount of data stored and transferred across the network and information about the Nodes on the network. The data on this page is automatically updated every hour so you can make time-series charts. + +You’ll also start seeing these statistics appear on various pages across the site, including our homepage and Node Operator page. These pages will be updated every hour when new data is published on the network statistics page. + + +The data we are exposing include the following statistics:  + + +### Statistics about stored and transferred data + +* **bandwidth\_bytes\_downloaded** - number of bytes downloaded (egress) from the network for the last 30 days +* **bandwidth\_bytes\_uploaded** - number of bytes uploaded (ingress) to the network for the last 30 days +* **storage\_inline\_bytes** - number of bytes stored in inline segments on the Satellite +* **storage\_inline\_segments** - number of segments stored inline on the Satellite +* **storage\_median\_healthy\_pieces\_count** - median number of healthy pieces per segment stored on Storage Nodes +* **storage\_min\_healthy\_pieces\_count** - minimum number of healthy pieces per segment stored on Storage Nodes +* **storage\_remote\_bytes** - number of bytes stored on Storage Nodes (it does not take into account the expansion factor of erasure encoding) +* **storage\_remote\_segments** - number of segments stored on Storage Nodes +* **storage\_remote\_segments\_lost** - number of irreparable segments lost from Storage Nodes +* **storage\_total\_bytes** - total number of bytes (both inline and remote) stored on the network +* **storage\_total\_object**s - total number of objects stored on the network +* **storage\_total\_pieces** - total number of pieces stored on Storage Nodes +* **storage\_total\_segments** - total number of segments stored on Storage Nodes +* **storage\_free\_capacity\_estimate\_bytes** - a statistical estimate of free Storage Node capacity, with suspicious values removed + +### Statistics about Storage Nodes + +* **active\_nodes** - number of Storage Nodes that were successfully contacted within the last 4 hours, excludes disqualified and exited Nodes +* **disqualified\_nodes** - number of disqualified Storage Nodes +* **exited\_nodes** - number of Storage Nodes that gracefully exited the Satellite, excludes disqualified Nodes +* **offline\_nodes** - number of Storage Nodes that were not successfully contacted within the last four hours, excludes disqualified and exited Nodes +* **suspended\_nodes** - number of suspended Storage Nodes, excludes disqualified and exited Nodes +* **total\_nodes** - total number of unique Storage Nodes that ever contacted the Satellite +* **vetted\_nodes** - number of vetted Storage Nodes, excludes disqualified and exited Nodes +* **full\_nodes** - number of Storage Nodes without free disk + +### Statistics about user accounts + +* **registered\_accounts** - number of registered user accounts + + + + +Since we launched this, one of our community members built this really cool [grafana dashboard](https://storjstats.info/d/storj/storj-network-statistics?orgId=1). Check it out. We’ll be sharing more about this and other community-built dashboards in the coming weeks, but we hope that exposing this data will continue to enable others to build amazing things like this! + + +As we continue to expand on the data points we expose, we’ll be adding more of this data to our [website](http://storj.io) as well. If you have any ideas or suggestions on what else we should be exposing, please open a GitHub [issue](https://github.com/storj/stats/issues) in the [repository](https://github.com/storj/stats) for this project. + + + + + diff --git a/app/(blog)/blog/our-3-step-interview-process-for-engineering-candidates/14f72bac7573e10c.png b/app/(blog)/blog/our-3-step-interview-process-for-engineering-candidates/14f72bac7573e10c.png new file mode 100644 index 000000000..cf3198eaf Binary files /dev/null and b/app/(blog)/blog/our-3-step-interview-process-for-engineering-candidates/14f72bac7573e10c.png differ diff --git a/app/(blog)/blog/our-3-step-interview-process-for-engineering-candidates/page.md b/app/(blog)/blog/our-3-step-interview-process-for-engineering-candidates/page.md new file mode 100644 index 000000000..84c4f6459 --- /dev/null +++ b/app/(blog)/blog/our-3-step-interview-process-for-engineering-candidates/page.md @@ -0,0 +1,151 @@ +--- +author: + name: JT Olio +date: '2019-03-25 00:00:00' +heroimage: ./14f72bac7573e10c.png +layout: blog +metadata: + description: In case you hadn't heard, Storj Labs is building a decentralized cloud + object storage service. Why would we do such a challenging thing? At a basic level, + it's because we believe the internet can be better than it currently is and we + see how to improve it. We believe your data is worse off being ... + title: Our 3-Step Interview Process for Engineering Candidates +title: Our 3-Step Interview Process for Engineering Candidates + +--- + +In case you hadn't heard, Storj Labs is building a decentralized cloud object storage service. Why would we do such a challenging thing? At a basic level, it's because we believe the internet can be better than it currently is and we see how to improve it. We believe your data is worse off being stored in the centralized data centers of five multinational mega-companies. + +Solving this grand problem requires us to solve many difficult sub-problems. Even though we are rigidly focusing on the simplest ways to solve these problems, the simplest solutions that can work in our space are still intensely complicated. To build our decentralized service, we need smart ideas and a capable team that isn’t afraid of solving challenges for the first time. If this sounds like fun to you, then you might be a good fit to join us on this adventure! + +Today I want to introduce you to our engineering interview process. + +### Hiring Values + +Before we talk about the details, it's important to list what we value, and what our goals are. We have two primary and co-equal objectives with recruiting: + +* Build the most diverse, inclusive, and welcoming team we can. +* Build the strongest technical team we can. + +We'll talk about being welcoming first. + +#### Diversity and Inclusion + +First off, pursuing a diverse team is the right thing to do, full stop. We recognize that the tech industry can be—and has been—an inaccessible place for people of underrepresented groups, so we value gate-opening (versus gate-keeping) wherever we can. + +Fortunately, a welcoming, inclusive, diverse team has a number of benefits that make it the clear choice even if it wasn't clearly the right thing to do! Monocultures breed all sorts of wacky pathologies, which is precisely the sort of thing we're fighting against technically with our decentralized storage platform! + +Did you know that [workforces with more gender, ethnic, or racial diversity tend to perform better financially](https://www.mckinsey.com/business-functions/organization/our-insights/why-diversity-matters)? According to McKinsey: + + +> Companies in the top quartile for racial and ethnic diversity are 35 percent more likely to have financial returns above their respective national industry medians. + +[According to Catalyst](https://www.catalyst.org/knowledge/bottom-line-corporate-performance-and-womens-representation-boards-20042008): + + +> Companies with sustained high representation of women board directors (WBD), defined as those with three or more WBD in at least four of five years, significantly outperformed those with sustained low representation by 84 percent on return on sales, by 60 percent on return on invested capital, and by 46 percent on return on equity. + +When you look at the hard data, it's clear that more diverse teams are more flexible, efficient, and effective. We want those benefits for our team too! + +As an aside, this is specifically a interviewing post, but diversity is not the end-goal of the journey. We said diversity and inclusion. Diversity is having many different backgrounds in your organization. Inclusion is having all of us feel welcome and wanting to be here. Literally centuries’ worth of company leadership advice talk about how important team morale and a sense of belonging is to all of us as human beings. + +So, it's not enough to just get employees through the door. To build a great team, you have to keep them! In a future blog, we’ll share more about what we're doing to support inclusion through our new Diversity & Inclusion Council. We’ll also share more about what diversity means to us because it’s more than just hiring more underrepresented people, such as women or LGBTQ employees. + +#### Technical aptitude + +To solve the hardest problems in decentralized storage, we absolutely need a world-class team. Great developers love learning, and therefore enjoy and seek out opportunities where they can learn from and grow thanks to others on their team. This means a world-class team tends to attract more world-class developers. Why do so many engineers want to work at, say, AmaGooSoft? It's not only about the money. + +Unfortunately, identifying which new recruits are world-class is very hard, and a lot of companies take lazy shortcuts. One such shortcut is hiring based on time-in-the-industry. Don't get me wrong, having more experience in distributed systems is a good thing for us, but requiring that someone has stuck around through the worst of our sometimes-polluted industry has too much of a filtering effect on our other top goal—a diverse and inclusive workplace. So, we need to find a more effective way to identify which candidates have top-shelf problem solving, communication, and programming skills, whether they have years of experience or are new to the industry. All backgrounds are welcome! + +In my experience, the best predictor of how successful someone will be at solving complex problems after six months on the job is their learning rate. We're specifically not looking for proficiency in a certain programming language, or knowledge of a specific skill, precisely because learning a language or skill should be the sort of thing our top candidates eat for breakfast. The best candidates are the sort of people who gravitate toward hard and confusing problems to tackle them and understand them, instead of flinching away from the discomfort of not knowing. We are building a decentralized cloud storage platform, after all. + +So, while we value experience, we also want to find candidates that demonstrate they are eager and adept at throwing themselves at hard problems they don't already know how to solve—and solving them. Great candidates are lifelong learners and this factors into our process. + +### Our process + +To achieve our two major values, we've had to make calls on a number of trade-offs. There are definitely downsides to our recruiting and interviewing process, but we feel that on-balance, the trade-offs are currently worth it. We're open to suggestions though! We’re constantly trying to learn and make improvements, and, considering interviewing ironically seems to have its own [Full Employment Theorem](https://en.wikipedia.org/wiki/Full_employment_theorem), we will probably be on this journey indefinitely. (Again, we believe in lifelong learning.) Given our values, if you can think of a way to help improve our process, please leave a comment or shoot us an [email](mailto:ask@storj.io)! + +#### Recruiting + +The recruiting stage of our pipeline is simply finding people who may want to come work for us. + +In an interview with Christiane Amanpour, Jon Stewart had this to say about diversity in comedy: + + +> It started out as a male-dominated field. It's not a particularly welcoming field—you sort of have to come out there and cut your teeth on it. I'll tell you a story: So we had, on The Daily Show, there was an article about us that said it was a sexist environment, we didn't have women writers. And I got very offended by that. I was very mad. I was like, "Are you saying I'm not a feminist?" I was raised by a single mother. She wore a T-shirt that said, "A woman needs a man like a fish needs a bicycle." And me and my brother were like, "I think we might be men?" So I was mad—how can they say such a thing? And I went back to the writers room, and I was like, "You believe this, Steve? What do you think, Greg? Dave? Tom? Mike?" And then I was like, Oooohhh. And it was right. + +But the reason it was right was not necessarily one that we had seen before. We had put in a system of getting writers where there were no names on it. We thought that's color-blind, gender-blind, et cetera. But what you don't realize is the system itself—the tributaries that feed us those submissions—is polluted itself… + + +> + +But do you see what I mean? It's a systemic issue, and I think what can mostly help change is when you open up new tributaries to bring in talent, and then they grow, and then they help grow their communities and tell their stories. + +Does this sound like the tech industry to you? It's not enough to simply say that you encourage candidates who might add diversity to your team to apply. You must find inputs to your recruiting funnel that represent greater diversity than the diversity in your field. It takes extra energy and effort. + +Like The Daily Show, the main technical evaluation stage of our interview process is name-blind. We are interested in hiring the best candidates based as much as possible on relevant qualifications alone. But Jon Stewart’s observation is a key insight to our recruiting process: if we only have a certain type of candidate, then all hires will be that same type of candidate! So while we don’t have quotas, we must continually analyze our incoming funnel of candidates to determine whether we are finding candidates with diverse backgrounds and experience. We are of course interested in hiring the most qualified candidate regardless of any other trait, but we also believe it is worth doing significant extra work to help build a diverse and inclusive workforce, which we believe will help Storj Labs be the best company it can be. This means that if our incoming pool of candidates does not represent these values, then we need to cast a wider net in our search. We reject the idea that diversity efforts are about lowering the bar--increasing diversity requires that we widen the net and include more people so we can raise the bar. + +We will continue to seek ways to improve our recruiting, and we welcome your thoughts on ways to make sure that everyone has an equal shot with Storj Labs. Again, if you have a suggestion on how we can improve, please let us know. + +#### Screening + +Each stage of our interview process takes work, so we want to quickly eliminate people from the process who aren't going to be the right fit for that position. + +In our screening stage, we want to ensure there won't be some sort of logistical problem. Is the candidate requesting more compensation than we have budgeted for the position? Is the candidate applying for the right job listing? Is the candidate lacking relevant job experience entirely? Where does the candidate expect to work, and will it require a relocation? Does the candidate seem communicative and not a jerk? Can we answer any questions? + +Hopefully most people sail on through this step. + +#### Name-blind homework problem + +This part of our interview process is the biggest, most time consuming, and potentially the most controversial, precisely because of how many trade-off decisions it makes. So, before we jump into what the downsides are, and why we do it anyway, let me just outline what we do. + +First, we invite candidates to an interview-specific Slack channel with a randomly chosen pseudonym. Originally we chose random animal species as part of the pseudonym, but we then switched to names of stars, because you're all stars! Interview candidates join the Slack channel to anonymously talk with the team. + +Second, the hiring manager posts a link to a homework problem in the channel. We are trying to select a problem that: + +1. is clear and concise, but with deep complexity +2. is fairly representative of day-to-day work +3. is considerably challenging, but preferably not due to the requirement of much additional existing knowledge +4. requires design discussion and architectural considerations prior to implementation +5. can be completed by our target candidates in under eight hours + +We don't set a deadline on the assignment because we want to be flexible with candidates' schedules. + +Third, we answer any questions the candidate has, discussing potential solutions and tradeoffs, and let the candidate get to work. + +Finally, we run the homework submission against tests and evaluate their assignment's code against a checklist. We pay $500 USD (in STORJ tokens) for problem solution submissions (whether or not they work completely). + +So, why do we do it this way? + +First, we want our interview to hinge on evaluating a candidate’s ability to do work in as real an environment as possible. We want them to use their IDE, their programming language, have access to documentation, relieve time pressure (if possible), and see how they communicate remotely (much of our own team is remote). We do our best to engage our candidates in a discussion about the problem; the Slack conversation is a big part of what we're looking for. So, we want to evaluate the candidate on work that is as much like real work as possible. + +Second, we use pseudonyms to let the candidate’s work stand for itself. At this point, we want this stage of our interview process to simply select the best possible candidates, independent of as many other factors as possible. This is how we use our focus on diversity to raise the bar—by including a wider pool of applicants, we can be even more selective at this stage. + +Third, we use a hard problem that isn’t completely specified. Just like real tickets we unfortunately file sometimes, the problem statement has assumptions that aren’t clear and require some level of additional requirements gathering and clarification. Just like international math tests, we want a problem that most people won’t ace. The greater the fidelity of the test, the more an excellent candidate will stand out. This, combined with our lack of specific experience criteria, is intended to allow inexperienced but sharp candidates the ability to shine. + +Fourth, we pay people. The major downside of our problem is it takes time, potentially time the candidate doesn't have. This is something we're pretty torn about. Unfortunately, a homework problem might eliminate people with busier home lives or who need to work elsewhere until they land the job with us, which is why we don’t set deadlines on the assignments. It's challenging to compress our hard problem into the span of an hour, but we're hopeful that compensating our candidates will help. + +Fifth, we try to grade as evenly and as routinely as possible. We have a checklist and we have a test harness. Even though we're already doing the interview at this stage name-blind, we still want to avoid as much bias as possible that might cause us to prefer anything besides what is explicitly stated as criteria in the homework problem description. + +#### Alternate option: name-blind technical work sample + +So, you’re probably thinking, “Some of the best candidates will most certainly balk at this huge hurdle you’re placing in their way.” To this we can only say, you’re right. Ultimately, we believe two things that make us pursue it anyway: we would rather have a process that occasionally rejects candidates that would have been a good fit than one that occasionally hires candidates that would have been a bad fit, and we believe great candidates are often just as interested in working on great teams (with stringent hiring criteria) as great teams are interested in hiring them. Many of our fantastic team members were more than willing to jump through this hurdle to join us! + +However, we do make a concession. For candidates that simply do not have the time for the assignment and will be forced to pass on a sweet job possibility with Storj Labs otherwise, we are happy to consider some other sample of work. If the candidate is able to provide us with some code sample they have created with sufficient complexity to be graded using our homework grading evaluation checklist, we will anonymize it and pass it to our review team. We will ask the candidate to explain the project to us in the anonymous Slack channel, so that we get a feel for how the candidate communicates. Of course, we expect that any candidates using this option will make sure that the sample is something that they have permission to share with us--please do not send us some third party’s intellectual property. Remember, you may not have the right to share a work sample with us even though you created the sample. Your work for another company likely belongs to that Company. We will reject any candidate who sends us third party IP that they do not have permission to share, or another person’s work as if it was their own. + +It’s a bit harder to grade these types of submissions evenly and fairly, so we are more picky with these types of submissions. + +#### Alternate option: white board interview + +If the rest of this post has failed to convince you of the benefits of our system, that’s okay! We are also willing to do a whiteboard interview with difficult algorithmic challenges. Let’s just say these won’t be [Fizz Buzz](https://blog.codinghorror.com/why-cant-programmers-program/)-style questions. These will be hard questions, and if you want to go this route (it is potentially the least time consuming), then we will be looking for you to dazzle us. We only leave this option available to be as flexible as possible, but be warned that we are the most picky for people who choose this option. We will certainly pass on some good candidates who choose this option. + +#### Team interview + +Our last phase is a team interview. This is just as much an opportunity for us as it is for the interview candidate. We want you to get to know the team! We want bidirectional question asking. Interview candidates should grill us on anything and everything (if they want). Otherwise, we’ll be asking performance-based interviewing questions. Sidenote: the U.S. Department of Veterans Affairs of all places has [the best collection of these we’ve found so far](https://www.va.gov/pbi/questions.asp)! + +Assuming no one finds any red flags in the team interview, the formal interview process is complete. We may have a few more follow-up questions, and you might as well. Sometimes, we must be forced to make hard choices between well-qualified candidates. But ideally, for candidates who make it this far, it’s time for onboarding, which is worth a separate blog post. + +### Final thoughts + +We’ve spent a lot of time thinking about and tweaking this process. Aside from the time our homework assignment takes to complete and grade, we feel like this is one of the better interview processes we’ve seen. Please [let us know](mailto:ask@storj.io) how we can improve! + diff --git a/app/(blog)/blog/page.js b/app/(blog)/blog/page.js new file mode 100644 index 000000000..efd269ee3 --- /dev/null +++ b/app/(blog)/blog/page.js @@ -0,0 +1,193 @@ +import React from 'react' +import Link from 'next/link' +import * as path from 'path' +import * as fs from 'fs' +import Markdoc from '@markdoc/markdoc' +import yaml from 'js-yaml' +import LocalImage from '@/components/LocalImage' + +export const metadata = { + title: 'Storj Engineering Blog', + description: + 'Learn about the latest developments in the Storj network and the technology that powers it.', + alternates: { + canonical: '/blog', + }, +} + +function getFrontmatter(filepath) { + const md = fs.readFileSync(filepath, 'utf8') + const ast = Markdoc.parse(md) + const frontmatter = ast.attributes.frontmatter + ? yaml.load(ast.attributes.frontmatter) + : {} + return { + title: frontmatter.title || path.basename(filepath), + frontmatter, + redirects: frontmatter.redirects, + docId: frontmatter.docId, + weight: frontmatter.weight, + } +} + +function sortByDateThenTitle(arr) { + arr.sort((a, b) => { + if (a.frontmatter.date !== b.frontmatter.date) { + return new Date(b.frontmatter.date) - new Date(a.frontmatter.date) + } else { + return a.title.localeCompare(b.title) + } + }) +} + +let dir = path.resolve('./app/(blog)') +function walkDirRec(dir, space) { + let results = [] + const list = fs.readdirSync(dir) + + list.forEach((file) => { + const filepath = path.join(dir, file) + const stat = fs.statSync(filepath) + const relativePath = path.join(file) + + if (stat && stat.isDirectory()) { + let pageFilepath = path.join(filepath, 'page.md') + // For directories that don't have an page.md + let title = file.charAt(0).toUpperCase() + file.slice(1) + let fm = null + if (fs.existsSync(pageFilepath)) { + fm = getFrontmatter(pageFilepath) + let entry = { + type: file, + title, + } + if (fm) { + entry = Object.assign(entry, fm) + } + if (fs.existsSync(pageFilepath)) { + entry.href = `/${space}/${relativePath}` + } + //if (fm.frontmatter.published) { + results.push(entry) + //} + } + } + }) + + return results +} + +let posts = walkDirRec(`${dir}/blog`, 'blog') +sortByDateThenTitle(posts) + +export default function BlogIndex() { + return ( + <> +
+
+
+

+ Storj Engineering Blog +

+

+ Learn about the latest developments in the Storj network and the + technology that powers it. +

+
+ {posts.map((post) => { + let frontmatter = post.frontmatter + return ( +
+ + {frontmatter.heroimage && ( +
+ +
+ )} +
+
+ + {/* + + {post.category.title} + +*/} +
+
+

+ + {post.frontmatter.title} +

+

+ {post.frontmatter.description || + post.frontmatter.metadata?.description || + post.excerpt} +

+
+ {/* +
+ +
+

+ + + {post.author.name} + +

+

{post.author.role}

+
+
+*/} +
+
+ {frontmatter.author && frontmatter.author.imageUrl && ( + + )} +
+

+ + {frontmatter.author.name} +

+

+ {frontmatter.author.role} +

+
+
+ +
+ ) + })} +
+
+
+
+ + ) +} diff --git a/app/(blog)/blog/production-concurrency/hero.jpeg b/app/(blog)/blog/production-concurrency/hero.jpeg new file mode 100644 index 000000000..2bff07d05 Binary files /dev/null and b/app/(blog)/blog/production-concurrency/hero.jpeg differ diff --git a/app/(blog)/blog/production-concurrency/page.md b/app/(blog)/blog/production-concurrency/page.md new file mode 100644 index 000000000..2a82d57dd --- /dev/null +++ b/app/(blog)/blog/production-concurrency/page.md @@ -0,0 +1,1097 @@ +--- +author: + name: Egon Elbre +date: '2022-07-29 00:00:00' +heroimage: ./hero.jpeg +layout: blog +metadata: + description: Concurrency is one of those things that's easy to get wrong, even with + Go concurrency features. Let's review things you should consider while writing + a concurrency production code. + title: Production Ready Go Concurrency +title: Production Ready Go Concurrency + +--- + +Concurrency is one of those things that's easy to get wrong, even with Go concurrency features. Let's review things you should consider while writing a concurrency production code. + +The guide is split into three parts, each with a different purpose. First, we'll talk about "Rules of Thumb," which are usually the right thing to do. The second part is on what to use for writing concurrent code. And finally, we'll cover how to write your custom concurrency primitives. + +Before we start, I should mention that many of these recommendations will have conditions where they are not the best choice. The main situations are going to be performance and prototyping. + +### Avoid Concurrency + +I've seen many times people using concurrency where you should not use it. It should go without saying, don't add concurrency unless you have a good reason. + +```go +var wg sync.WaitGroup + +wg.Add(1) +go serve(&wg) +wg.Wait() +``` + +❯ + +```go +serve() +``` + +The concurrency here is entirely unnecessary, but I've seen this exact code in a repository. System without concurrency is much easier to debug, test and understand. + +People also add concurrency because they think it will speed up their program. In a production environment, you are handling many concurrent requests anyways, so making one part concurrent doesn't necessarily make the whole system faster. + +### Prefer Synchronous API + +A friend of the previous rule is to prefer synchronous API. As mentioned, non-concurrent code is usually shorter and easier to test and debug. + + +```go +server.Start(ctx) +server.Stop() +server.Wait() +``` + +❯ + +```go +server.Run(ctx) +``` + +If you need concurrency when using something, it's relatively easy to make things concurrent. It's much more difficult to do the reverse. + +### Use -race and t.Parallel() + +There are two excellent Go features that help you shake out concurrency bugs from your code. + +First is -race, which enables the race detector to flag all the observed data races. It can be used with go test -race ./... or go build -race ./yourproject. See [Data Race Detector](https://go.dev/doc/articles/race_detector) for more details. + +Second mark your tests with t.Parallel(): + +```go +func TestServer(t *testing.T) { + t.Parallel() + // ... +``` +This makes your tests run in parallel, which can speed them up, but it also means you are more likely to find a hidden shared state that doesn't work correctly in concurrent code. In addition to finding bugs in our codebases, we've also found them in third-party libraries. + +### No global variables + +Avoid global variables such as caches, loggers, and databases. + +For example, it's relatively common for people to use log.Println inside their service, and their testing output ends in the wrong location. + + +```go +func TestAlpha(t *testing.T) { + t.Parallel() + log.Println("Alpha") +} + +func TestBeta(t *testing.T) { + t.Parallel() + log.Println("Beta") +} +``` + +The output from go test -v will look like: + +``` +=== RUN TestAlpha +=== PAUSE TestAlpha +=== RUN TestBeta +=== PAUSE TestBeta +=== CONT TestAlpha +=== CONT TestBeta +2022/07/24 10:59:06 Alpha +--- PASS: TestAlpha (0.00s) +2022/07/24 10:59:06 Beta +--- PASS: TestBeta (0.00s) +PASS +ok test.test 0.213s +``` + +Notice how the "Alpha" and "Beta" are out of place. The code under test should call t.Log for any testing needs; then, the log lines will appear in the correct location. There's no way to make it work with a global logger. + +### Know when things stop + +Similarly, it's relatively common for people to start goroutines without waiting for them to finish. *go* keyword makes starting goroutines very easy; however, it's not apparent that you also must wait for them to stop. + + +```go +go ListenHTTP(ctx) +go ListenGRPC(ctx) +go ListenDebugServer(ctx) +select{} +``` + +❯ + +```go +g, ctx := errgroup.WithContext(ctx) +g.Go(func() error { + return ListenHTTP(ctx) +} +g.Go(func() error { + return ListenGRPC(ctx) +} +g.Go(func() error { + return ListenDebugServer(ctx) +} +err := g.Wait() +``` + +When you don't know when things stop, you don't know when to close your connections, databases, or log files. For example, some stray goroutine might use a closed database and cause panic. + +Similarly, when you wait for all goroutines to finish, you can detect scenarios when one of the goroutines has become indefinitely blocked. + +### Context aware code + +The next common issue is not handling context cancellation. It usually won't be a problem in the production system itself. It's more of an annoyance during testing and development. Let's imagine you have a time.Sleep somewhere in your code: + + +```go +time.Sleep(time.Minute) +``` + +❯ + +```go +tick := time.NewTimer(time.Minute) +defer tick.Stop() + +select { +case <-tick.C: +case <-ctx.Done(): + return ctx.Err() +} +``` + +time.Sleep cannot react to any code, which means when you press Ctrl-C on your keyboard, it will stay on that line until it finishes. This can increase your test times due to some services shutting down slowly. Or, when doing upgrades on your servers, it can make them much slower to shut down. + +*The code for the waiting on the right is much longer, but we can write helpers to simplify it.* + +The other scenario where this cancellation comes up is long calculations: + + +```go +for _, f := range files { + data, err := os.ReadFile(f) + // ... +} +``` + +❯ + +```go +for _, f := range files { + if err := ctx.Err(); err != nil { + return err + } + + data, err := os.ReadFile(f) + // ... +} +``` + +Here we can introduce a ctx.Err() call to check whether the context has been cancelled. Note ctx.Err() call is guaranteed to be concurrency safe, and it's not necessary to check ctx.Done() separately. + +### No worker pools + +People coming from other languages often resort to creating worker pools. It's one of those tools that's necessary when you are working with threads instead of goroutines. + +There are many reasons to not use worker pools: + +* They make stack traces harder to read. You'll end up having hundreds of goroutines that are on standby. +* They use resources even if they are not working. +* They can be slower than spawning a new goroutine. + +You can replace your worker pools with a goroutine limiter -- something that disallows from creating more than N goroutines. + +```go +var wg sync.WaitGroup +defer wg.Wait() +queue := make(chan string, 8) +for k := 0; k < 8; k++ { + wg.Add(1) + go func() { + defer wg.Done() + for work := range queue { + process(work) + } + }() +} + +for _, work := range items { + queue <- work +} +close(queue) +``` + +❯ + +```go +var wg sync.WaitGroup +defer wg.Wait() +limiter := make(chan struct{}, 8) +for _, work := range items { + work := work + wg.Add(1) + limiter <- struct{}{} + go func() { + defer wg.Done() + defer func() { <-limiter }() + + process(work) + }() +} +``` +We'll later show how to make a limiter primitive easier to use. + +### No polling + +Polling another system is rather wasteful of resources. It's usually better to use some channel or signal to message the other side: + +```go +lastKnown := 0 +for { + time.Sleep(time.Second) + t.mu.Lock() + if lastKnown != t.current { + process(t.current) + lastKnown = t.current + } + t.mu.Unlock() +} +``` + +❯ + +```go +lastKnown := 0 +for newState := range t.updates { + if lastKnown != newState { + process(newState) + lastKnown = newState + } +} +``` + +Polling wastes resources when the update rates are slow. It also responds to changes slower compared to notifying directly. There are many ways to avoid polling, which could be a separate article altogether. + +*Of course, if you are making an external request and the external API is out of your control, you might not have any other choice than to poll.* + +### Defer unlocks and waits + +It's easy to forget an mu.Unlock, wg.Wait or close(ch). If you always defer them, it will be much easier to see when they are missing. + + +```go +for _, item := range items { + service.mu.Lock() + service.process(item) + service.mu.Unlock() +} +``` + +❯ + +```go +for _, item := range items { + func() { + service.mu.Lock() + defer service.mu.Unlock() + + service.process(item) + }() +} +``` + +Even if your initial code is correct, then code modification can introduce a bug. For example, adding a return inside the loop after the mu.Lock() would leave the mutex locked. + +### Don’t expose your locks + +The larger the scope where the locks can be used, the easier it is to make a mistake. + + +```go +type Set[T any] struct { + sync.Lock + Items []T +} +``` + +❯ + +```go +type Set[T any] struct { + mu sync.Lock + items []T +} +``` + +### Name your goroutines + +You can make your debugging and stack traces much nicer by adding names to your goroutines: + +```go +labels := pprof.Labels("server", "grpc") +pprof.Do(ctx, labels, + func(ctx context.Context) { + // ... + }) +``` + +There's an excellent article "[Profiler labels in Go](https://rakyll.org/profiler-labels/)", which explains how to use them. + +## Concurrency Primitives + +When it comes to writing production code, it's a bad idea to use some concurrency primitives directly in your code. They can be error-prone and make code much harder to reason about. + +When choosing primitives, prefer them in this order: + +1. no-concurrency +2. golang.org/x/sync/errgroup, golang.org/x/sync, sync.Once +3. custom primitive or another library +4. sync.Mutex in certain scenarios +5. select { + +However, many others are useful when used for implementing your custom primitives: + +5. sync.Map, sync.Pool (use a typesafe wrapper) +6. sync.WaitGroup +7. chan, go func() { +8. sync.Mutex, sync.Cond +9. sync/atomic + +If you are surprised that chan and go func() { are so low on the list, we'll show how people make tiny mistakes with them. + +### Common Mistake #1: go func() + +```go +func (server *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { + ... + // start an async operation + go func() { + res, err := server.db.ExecContext(r.Context(), "INSERT ...") + ... + }() + ... +} + +func main() { + ... + + db, err := openDB(ctx) + defer db.Close() + + err := server.Run(ctx) + ... +} +``` + +Notice there's no guarantee that the goroutine finishes before the database is closed. This can introduce weird test failure, where you try to insert into a closed database. + +Similarly, another bug, r.Context() could be cancelled prematurely. Of course, this depends on the problem specifics, but when you start a background operation from your handler, you don't want the user to cancel it. + +### Primitive: sync.WaitGroup + +One of the solutions for starting goroutines is to use sync.WaitGroup. However, it also has quite a few problematic scenarios. + +Let's take a look at the first common mistake with sync.WaitGroup: + + +```go +func processConcurrently(item []*Item) { + var wg sync.WaitGroup + defer wg.Wait() + for _, item := range items { + item := item + go func() { + process(&wg, item) + }() + } +} + +func process(wg *sync.WaitGroup, item *Item) { + wg.Add(1) + defer wg.Done() + + ... +} +``` + +Here the problem is that the processConcurrently can return before wg.Add is called. This means that we don't wait for all the goroutines to finish. + +The other scenario comes up when people incrementally change code: + +```go +func processConcurrently(item []*Item) { + var wg sync.WaitGroup + wg.Add(len(items)) + defer wg.Wait() + for _, item := range items { + item := item + if filepath.Ext(item.Path) != ".go" { + continue + } + go func() { + defer wg.Done() + process(item) + }() + } +} +``` + +Notice how we moved the call to wg.Done outside of the process, making it easier to track the full concurrency. However, due to the extra if filepath.Ext statement, the code is wrong. That check was probably added by someone else at a later time. Similarly, it's one of those cases where tests might easily miss the problem. + +To fully fix the code, it should look like this: + +```go +func processConcurrently(item []*Item) { + var wg sync.WaitGroup + defer wg.Wait() + for _, item := range items { + item := item + if filepath.Ext(item.Path) != ".go" { + continue + } + wg.Add(1) + go func() { + defer wg.Done() + process(item) + }() + } +} +``` + +If you don't see the following parts when someone is using sync.WaitGroup, then it probably has a subtle error somewhere: + +```go +var wg sync.WaitGroup +defer wg.Wait() +... +for ... { + wg.Add(1) + go func() { + defer wg.Done() +``` + +### Use golang.org/x/sync/errgroup + +Instead of sync.WaitGroup there's a better alternative that avoids many of these issues: + +```go +func processConcurrently(item []*Item) error { + var g errgroup.Group + for _, item := range items { + item := item + if filepath.Ext(item.Path) != ".go" { + continue + } + g.Go(func() error { + return process(item) + }) + } + return g.Wait() +} +``` + +errgroup.Group can be used in two ways: + +```go +// on failure, waits other goroutines +// to stop on their own +var g errgroup.Group +g.Go(func() error { + return publicServer.Run(ctx) +}) +g.Go(func() error { + return grpcServer.Run(ctx) +}) +err := g.Wait() +``` + +```go +// on failure, cancels other goroutines +g, ctx := errgroup.WithContext(ctx) +g.Go(func() error { + return publicServer.Run(ctx) +}) +g.Go(func() error { + return grpcServer.Run(ctx) +}) +err := g.Wait() +``` + +You can read [golang.org/x/sync/errgroup documentation](https://pkg.go.dev/golang.org/x/sync/errgroup#Group) for additional information. *Note, errgroup allows to limit the number of goroutines that can be started concurrently.* + +### Primitive: sync.Mutex + +Mutex is definitely a useful primitive, however you should be careful when you use it. I've seen quite often code that looks like: + + +```go +func (cache *Cache) Add(ctx context.Context, key, value string) { + cache.mu.Lock() + defer cache.mu.Unlock() + + cache.evictOldItems() + cache.items[key] = entry{ + expires: time.Now().Add(time.Second), + value: value, + } +} +``` + +You might wonder, what's the problem here. It's appropriately locking and unlocking. The main problem is the call to cache.evictOldItemsand that it's not handling context cancellation. This means that requests could end up blocking behind cache.mu.Lock, and even if they are cancelled you would need to wait for it to get unlocked before you can return. + +Instead, you can use a chan \*state, which allows you to handle context cancellation properly: + +```go +type Cache struct { + state chan *state +} + +func NewCache() { + content := make(chan *state, 1) + content <- &state{} + return Cache{state: content} +} + +func (cache *Cache) Add(ctx context.Context, key, value string) error { + select { + case <-ctx.Done(): + return ctx.Err() + case state := <-cache.state: + defer func() { cache.state <- state }() + + cache.evictOldItems() + cache.items[key] = entry{ + expires: time.Now().Add(time.Second), + value: value, + } + + return nil + } +} +``` + +Even though the evictOldItems call is still there, it won't prevent other callers to Add to cancel their request. + +Use sync.Mutex only for cases where you need to hold the lock for a short duration. Roughly it means that the code is O(N) or better, and N is small. + +#### Primitive: sync.RWMutex + +sync.RWMutex has all the same problems as sync.Mutex. However, it can also be significantly slower. Similarly, it makes it easy to have data races when you write to variables during RLock. + +In your specific scenario, you should have benchmarks demonstrating that sync.RWMutex is faster than sync.Mutex. + +*Details: When there are a lot of readers and no writers, there's a cache contention between the readers because taking a read lock mutates a mutex, which is not scalable. A writer attempting to grab the lock blocks future readers from acquiring it, so long-lived readers with infrequent writers cause long delays of no work.* + +Either way, you should be able to demonstrate that your use of sync.RWMutex is helpful. + +### Primitive: chan + +Channels are valuable things in the Go language but are also error-prone. There are many ways to write bugs with them: + + +```go +const workerCount = 100 + +var wg sync.WaitGroup +workQueue := make(chan *Item) +defer wg.Wait() + +for i := 0; i < workerCount; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for item := range workQueue { + process(item) + } + }() +} + +err := db.IterateItems(ctx, func(item *Item) { + workQueue <- item +}) +``` + +This is probably one of the common ones... forgetting to close the channel. Channels also make the code harder to review compared to using higher-level primitives. + +Using chan for communicating between different "goroutine processes" in your application is fine; however, ensure that you handle context cancellations and shut down properly. Otherwise, it's easy to introduce a deadlock. + +### Few additional rules-of-thumb + +I've come to the conclusion that you should avoid these in your domain logic: + +* make(chan X, N) +* go func() +* sync.WaitGroup + +They are error-prone, and there are better approaches. It's clearer to write your own higher-level abstraction for your domain logic. Of course, having them isn't an "end-of-the-world" issue either. + +I should separately note that using "select" is usually fine. + +## Your own artisanal concurrency primitives + +I told you to avoid many things in domain code, so what should you do instead? + +If you cannot find an appropriate primitive from golang.org/x/sync or other popular libraries... you can write your own. + +> Writing a separate concurrency primitive is easier to get right than writing ad hoc concurrency logic in domain code. + +There are many ways you can write such primitives. The following are merely examples of different ways how you can write them. + +### Sleeping + +Let's take a basic thing first, sleeping a bit: + +```go +func Sleep(ctx context.Context, duration time.Duration) error { + t := time.NewTimer(duration) + defer t.Stop() + + select { + case <-t.C: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} +``` +Here we need to ensure that we appropriately react to context cancellation so that we don't wait for a long time until we notice that context canceled the operation. Using this call is not much longer than time.Sleep itself: + + +```go +if err := Sleep(ctx, time.Second); err != nil { + return err +} +``` +### Limiter + +I've found plenty of cases where you must limit the number of goroutines. + + +```go +type Limiter struct { + limit chan struct{} + working sync.WaitGroup +} + +func NewLimiter(n int) *Limiter { + return &Limiter{limit: make(chan struct{}, n)} +} + +func (lim *Limiter) Go(ctx context.Context, fn func()) bool { + // ensure that we aren't trying to start when the + // context has been cancelled. + if ctx.Err() != nil { + return false + } + + // wait until we can start a goroutine: + select { + case lim.limit <- struct{}{}: + case <-ctx.Done(): + // maybe the user got tired of waiting? + return false + } + + lim.working.Add(1) + go func() { + defer func() { + <-lim.limit + lim.working.Done() + }() + + fn() + }() + + return true +} + +func (lim *Limiter) Wait() { + lim.working.Wait() +} +``` + +This primitive is used the same way as errgroup.Group: + +```go +lim := NewLimiter(8) +defer lim.Wait() +for _, item := range items { + item := item + started := lim.Go(ctx, func() { + process(item) + }) + if !started { + return ctx.Err() + } +} +``` + +Of course, if your limited goroutines are dependent on each other, then it can introduce a deadlock. + +> Note that there's a potential "bug" with using such a Limiter. You must not call limiter.Go after you have called limiter.Wait, otherwise the goroutine can be started after limiter.Wait has returned. This can also happen with sync.WaitGroup and errgroup.Group. One way to avoid this problem is to disallow starting goroutines after limiter.Wait has been called. It probably makes sense to rename it to "limiter.Close" in that case. + +#### Batch processing a slice + +Let's say you want to process a slice concurrently. We can use this limiter to start multiple goroutines with the specified batch sizes: + + +```go +type Parallel struct { + Concurrency int + BatchSize int +} + +func (p Parallel) Process(ctx context.Context, + n, process func(low, high int)) error { + + // alternatively, these panics could set a default value + if p.Concurrency <= 0 { + panic("concurrency must be larger than zero") + } + if p.BatchSize <= 0 { + panic("batch size must be larger than zero") + } + + lim := NewLimiter(p.Concurrency) + defer lim.Wait() + + for low := 0; low < n; low += p.BatchSize { + low, high := low, low + p.BatchSize + if high > n { + high = n + } + + started := lim.Go(ctx, func() { + process(low, high) + }) + if !started { + return ctx.Err() + } + } +} +``` + +This primitive allows to hide the "goroutine management" from our domain code: + +```go +var mu sync.Mutex +total := 0 + +err := Parallel{ + Concurrency: 8, + BatchSize: 256, +}.Process(ctx, len(items), func(low, high int) { + price := 0 + for _, item := range items[low:high] { + price += item.Price + } + + mu.Lock() + defer mu.Unlock() + total += price +}) +``` + +### Running a few things concurrently + +Sometimes for testing, you need to start multiple goroutines and wait for all of them to complete. You can use errgroup for it; however, we can write a utility that makes it shorter: + + +```go +func Concurrently(fns ...func() error) error { + var g errgroup.Group + for _, fn := range fns { + g.Go(fn) + } + return g.Wait() +} +``` + +A test can use it this way: + +```go +err := Concurrently( + func() error { + if v := cache.Get(123); v != nil { + return errors.New("expected value for 123") + } + return nil + }, + func() error { + if v := cache.Get(256); v != nil { + return errors.New("expected value for 256") + } + return nil + }, +) +if err != nil { + t.Fatal(err) +} +``` + +There are many variations of this. Should the function take ctx as an argument and pass it to the child goroutines? Should it cancel all the other functions via context cancellations when one error occurs? + +### Waiting for a thing + +Sometimes you want different goroutines to wait for one another: + +```go +type Fence struct { + create sync.Once + release sync.Once + wait chan struct{} +} + +// init allows to use the struct without separate initialization. +func (f *Fence) init() { + f.create.Do(func() { + f.wait = make(chan struct{}) + }) +} + +// Release releases any waiting goroutines. +func (f *Fence) Release() { + f.init() + f.release.Do(func() { + close(f.wait) + }) +} + +// Released allows to write different select than +// `Fence.Wait` provides. +func (f *Fence) Released() chan struct{} { + f.init() + return f.wait +} + +// Wait waits for the fence to be released and takes into account +// context cancellation. +func (f *Fence) Wait(ctx context.Context) error { + f.init() + select { + case <-f.Released(): + return nil + case <-ctx.Done(): + return ctx.Err() + } +} +``` + +When we use it together with Concurrently we can write code that looks like: + +```go +var loaded Fence +var data map[string]int + +err := Concurrently( + func() error { + defer loaded.Release() + data = getData(ctx, url) + return nil + }, + func() error { + if err := loaded.Wait(ctx); err != nil { + return err + } + return saveToCache(data) + }, + func() error { + if err := loaded.Wait(ctx); err != nil { + return err + } + return processData(data) + }, +) +``` + +### Protecting State + +Similarly, we quite often need to protect the state when concurrently modifying it. We've seen how sync.Mutex is sometimes error-prone and doesn't consider context cancellation. Let's write a helper for such a scenario. + + +```go +type Locked[T any] struct { + state chan *T +} + +func NewLocked[T any](initial *T) *Locked[T] { + s := &Locked[T]{} + s.state = make(chan *T, 1) + s.state <- initial + return s +} + +func (s *Locked[T]) Modify(ctx context.Context, fn func(*T) error) error { + if ctx.Err() != nil { + return ctx.Err() + } + + select { + case state := <-s.state: + defer func() { s.state <- state }() + return fn(state) + case <-ctx.Done(): + return ctx.Err() + } +} +``` + +Then we can use it like: + +```go +state := NewLocked(&State{Value: 123}) +err := state.Modify(ctx, func(state *State) error { + state.Value = 256 + return nil +}) +``` + +### Async processes in a server + +Finally, let's take a scenario where we want to start background goroutines inside a server. + +Let's first write out the server code, how we would like to use it: + +```go +func (server *Server) Run(ctx context.Context) error { + server.pending = NewJobs(ctx) + defer server.pending.Wait() + + return server.listenAndServe(ctx) +} + +func (server *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { + ... + + started := server.pending.Go(r.Context(), + func(ctx context.Context) { + err := server.db.ExecContext(ctx, "INSERT ...") + ... + }) + if !started { + if r.Context().Err() != nil { + http.Error(w, "client closed request", 499) + return + } + http.Error(w, "shutting down", http.StatusServiceUnavailable) + return nil + } + + ... +} +``` + +Then let's write the primitive: + +```go +type Jobs struct { + root context.WithContext + group errgroup.Group +} + +func NewJobs(root context.Context) *Jobs { + return &Jobs{root: root} +} + +func (jobs *Jobs) Wait() { _ = jobs.group.Wait() } + +func (jobs *Jobs) Go(requestCtx context.Context, fn func(ctx context.Context)) bool { + // did the user cancel? + if requestCtx.Err() != nil { + return false + } + // let's check whether server is shutting down + if jobs.root.Err() != nil { + return false + } + + jobs.group.Go(func() error { + // Note, we use the root context and not the request context. + fn(jobs.root) + return nil + }) + + return true +} +``` + +Of course, we can add a limiter, to prevent too many background workers to be started: + +```go +type Jobs struct { + root context.WithContext + limit chan struct{} + group errgroup.Group +} + +func (jobs *Jobs) Go(requestCtx context.Context, fn func(ctx context.Context)) bool { + if requestCtx.Err() != nil || jobs.root.Err() != nil { + return false + } + select { + case <-requestCtx.Done(): + return false + case <-jobs.root.Done(): + return false + case jobs.limit <- struct{}{}: + } + + jobs.group.Go(func() error { + defer func() { <-jobs.limit }() + fn(ctx) + return nil + }) + + return true +} +``` + +### Exercise: Retrying with backoff + +As a final exercise for the reader, you can try implementing a retry with backoff. The API for such a primitive can look like this: + +```go +const ( + maxRetries = 10 + minWait = time.Second/10 + maxWait = time.Second +) + +retry := NewRetry(maxRetries, minWait, maxWait) +for retry.Next(ctx) { + ... +} +if retry.Err() != nil { + return retry.Err() +} +``` + +Alternatively, it can be callback based: + +```go +err := Retry(ctx, maxRetries, minWait, maxWait, + func(ctx context.Context) error { + ... + }) +``` + +Additionally, consider where one would be better than the other. + +## Additional resources + +There are many resources that can help you delve deeper. + +You can find quite a lot of **our own custom primitives** at [**storj.io/common/sync2**](https://pkg.go.dev/storj.io/common/sync2). This package contains most of our synchronization primitives. It contains things like *Sleep* and *Concurrently*, but also more advanced things like *Cycle*, *ReadCache* and *Throttle*. We also have problem specific implementations of [**Combiner**](https://github.com/storj/storj/blob/main/satellite/metainfo/piecedeletion/combiner.go#L15) and [**Queue**](https://github.com/storj/storj/blob/6df867bb3d06240da139de145aaf88077572b4b8/satellite/metainfo/piecedeletion/queue.go#L10) that implement a combiner queue. This primitive allows to dial storage nodes, coalesce multiple deletion requests into a single request. + +One of the best talks about Go concurrency is "[**Rethinking Classical Concurrency Patterns**](https://www.youtube.com/watch?v=5zXAHh5tJqQ)" by **Bryan C. Mills**. He discusses problems with worker pools and sync.Cond in-depth. + +When you struggle with understanding data-races, then "[**Little Book of Semaphores**](https://greenteapress.com/wp/semaphores/)" by **Allen B. Downey** is an excellent resource. It contains many classic problems and exercises to get your brain noticing them. + +There has been also some research on the topic "[**Real-World Concurrency Bugs in Go**](https://songlh.github.io/paper/go-study.pdf)" by **Tengfei Tu** et. al. It contains many additional issues not mentioned in this post. diff --git a/app/(blog)/blog/storj-open-development-announcement/10fb6f750e522e1c.png b/app/(blog)/blog/storj-open-development-announcement/10fb6f750e522e1c.png new file mode 100644 index 000000000..9b6dd8fe6 Binary files /dev/null and b/app/(blog)/blog/storj-open-development-announcement/10fb6f750e522e1c.png differ diff --git a/app/(blog)/blog/storj-open-development-announcement/page.md b/app/(blog)/blog/storj-open-development-announcement/page.md new file mode 100644 index 000000000..55232b408 --- /dev/null +++ b/app/(blog)/blog/storj-open-development-announcement/page.md @@ -0,0 +1,67 @@ +--- +author: + name: Clement Sam +date: '2021-10-01 00:00:00' +heroimage: ./10fb6f750e522e1c.png +layout: blog +metadata: + description: Storj Open Development Announcement + title: Storj Open Development Announcement +title: Storj Open Development Announcement + +--- + + + + +As you all know, all our code for Storj V3 is [open source](https://github.com/storj/storj). Our team believes that openness and transparency are critical to everything we do at Storj.  We also value feedback from our Storage Node Operators and the community as a whole. Today we’re announcing that we intend to adopt an open development strategy for our Storage Node development to ensure that everyone can get involved in developing the Storage Nodes or contribute to the network. + + + + +As part of this, we’ve moved all our Storage Node Operator Jira tickets to GitHub, and new issues will be tracked in the [Storage Node Project Board](https://github.com/orgs/storj/projects/6#card-69201765) on GitHub. + + + + +Our goal is to do this in a way that embraces the well-established open source model that’s been working effectively for years: meaningful and positive contributions that align to long-standing, thoughtfully designed architecture and collaborative engineering. Together we seek the best outcome for all people who use Storj. + + + + +**Who can contribute?** + + + + +Everyone - Developers interested in contributing to the Storj Open Source Project are invited to contribute code or open issues with bug reports and feature requests.  + + + + +Contributions are not only restricted to issues with bug reports or feature requests. We’re open to pull requests in the form of bug fixes, tests, new features, UI enhancements, or bug reports in the form of PR with a failing test. Any pull requests that help the network and improve upon our open source software are welcome, reviewed, and accepted into the Storj platform. + + + + +Our primary focus for the Storage Node development is the multi-node dashboard. Any improvement is welcome. If you’re not running a Storage Node yourself, we can send you a list of nodeIDs that you can add to your dashboard. You can start with a multi-node dashboard filled with several months of data. + + + + +We also look forward to pull requests with UI tests with [go-rod](https://github.com/go-rod/rod).  + + + + +**What happens next?** + +If you’re a Storage Node Operator, a Storj customer, an open source enthusiast, or a developer interested in contributing to the Storj network, [we invite you to collaborate](https://github.com/storj/storj/contribute) and bring the best of Storj forward to continue to make the internet decentralized and secure for everyone. + +**Helpful links:** + +* **Getting Started**: +* **Setting up a local instance of the storj network using storj-sim**: + +**Storj V3 whitepaper**: + diff --git a/app/(blog)/blog/storj-open-development-part-2-whats-new/69c628262ae22ee5.png b/app/(blog)/blog/storj-open-development-part-2-whats-new/69c628262ae22ee5.png new file mode 100644 index 000000000..8c830c24b Binary files /dev/null and b/app/(blog)/blog/storj-open-development-part-2-whats-new/69c628262ae22ee5.png differ diff --git a/app/(blog)/blog/storj-open-development-part-2-whats-new/75072fcbffc78ed1.png b/app/(blog)/blog/storj-open-development-part-2-whats-new/75072fcbffc78ed1.png new file mode 100644 index 000000000..91cf61d86 Binary files /dev/null and b/app/(blog)/blog/storj-open-development-part-2-whats-new/75072fcbffc78ed1.png differ diff --git a/app/(blog)/blog/storj-open-development-part-2-whats-new/page.md b/app/(blog)/blog/storj-open-development-part-2-whats-new/page.md new file mode 100644 index 000000000..44d2780c4 --- /dev/null +++ b/app/(blog)/blog/storj-open-development-part-2-whats-new/page.md @@ -0,0 +1,56 @@ +--- +author: + name: Brandon Iglesias +date: '2022-03-31 00:00:00' +heroimage: ./69c628262ae22ee5.png +layout: blog +metadata: + description: "In October 2021, Storj announced we were going to adopt an open development\ + \ strategy for the storage node development efforts. The goal was to enable our\ + \ community\u2014and the wider open source community\u2014to contribute to the\ + \ development of the network. We started this effort by moving all node issues..." + title: Storj Open Development -Part 2 +title: Storj Open Development -Part 2 + +--- + +In October 2021, Storj [announced](https://www.storj.io/blog/storj-open-development-announcement) we were going to adopt an open development strategy for the storage node development efforts. The goal was to enable our community—and the wider open source community—to contribute to the development of the network. We started this effort by moving all node issues to the [storj Github repo](https://github.com/storj/storj/issues?q=is%3Aopen+is%3Aissue+label%3ASNO) and creating a public GitHub [project](https://github.com/orgs/storj/projects/6) to track them. This allows anyone in the [community](https://forum.storj.io/) to look at current and past issues, add comments, make code contributions, or open new issues. + + + +Over the last six months, our engineering and product teams have been adopting this open development strategy for all our efforts. Our teams have moved from private Jira tickets to Github issues/projects to create more transparency with what we are working on and allow the community to make contributions. + + + +In addition, we have made the Storj Network product road map public. The [product road map](https://github.com/storj) is a high-level overview of the features and functionality we plan on implementing over the next nine - twelve months. It is constantly evolving as we gain more input and insights from our customers and community, so keep in mind road map items are subject to change. + + + +As a part of this open development initiative, before starting development the product team will now make product requirements documents (PRD) public, describing the problem/challenge we intend to solve for our customers. The engineering team uses the PRD to create a blueprint on how we intend to implement the functionality, and our QA team will develop a test plan. Through each of these steps, the documents will be published in our GitHub repositories where reviews and input from our community and customers are welcome and very much appreciated. + + + +![](./75072fcbffc78ed1.png) + +Team boards + +* Storj Network road map: +* Metainfo team: +* Edge team: +* Documentation: + + + +\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ + +TLDR: + +We launched our open development strategy in October and have been making significant progress. Six months later, check out some of our initiatives: + +* Moved issue tracking to Github where it's open and transparent to everyone +* Created the [Storj Network public road map](https://github.com/orgs/storj/projects/23) also on Github +* Continuing our [Bug bounty program](#) +* Looking for more contributions from the community + + + diff --git a/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/06a4d2d7af413c6a.png b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/06a4d2d7af413c6a.png new file mode 100644 index 000000000..7d3edceed Binary files /dev/null and b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/06a4d2d7af413c6a.png differ diff --git a/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/344a9e476aa2b1ea.png b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/344a9e476aa2b1ea.png new file mode 100644 index 000000000..d9267dee2 Binary files /dev/null and b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/344a9e476aa2b1ea.png differ diff --git a/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/4b79f2e770056a31.png b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/4b79f2e770056a31.png new file mode 100644 index 000000000..a4f4e53b3 Binary files /dev/null and b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/4b79f2e770056a31.png differ diff --git a/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/6de1c07a519802ed.png b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/6de1c07a519802ed.png new file mode 100644 index 000000000..5c5663386 Binary files /dev/null and b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/6de1c07a519802ed.png differ diff --git a/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/943dcd25f919a98b.png b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/943dcd25f919a98b.png new file mode 100644 index 000000000..db58bd8cf Binary files /dev/null and b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/943dcd25f919a98b.png differ diff --git a/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/c911680feffbe653.png b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/c911680feffbe653.png new file mode 100644 index 000000000..d9dcf7361 Binary files /dev/null and b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/c911680feffbe653.png differ diff --git a/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/e76c6da3bc2247e7.png b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/e76c6da3bc2247e7.png new file mode 100644 index 000000000..ee4454123 Binary files /dev/null and b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/e76c6da3bc2247e7.png differ diff --git a/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/page.md b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/page.md new file mode 100644 index 000000000..69120313d --- /dev/null +++ b/app/(blog)/blog/use-storj-dcs-from-cloud-native-environments-using-sidecar-pattern/page.md @@ -0,0 +1,213 @@ +--- +author: + name: Marton Elek +date: '2022-03-07 00:00:00' +heroimage: ./344a9e476aa2b1ea.png +layout: blog +metadata: + description: "Data stored in Storj Decentralized Cloud Storage can be accessed in\ + \ multiple ways:With native \u201Cuplink\u201D protocol, which connects directly\ + \ to the nodes where the data is storedWith using S3 compatible REST API, using\ + \ an S3 gateway:Either the hosted S3 gateway, operated by Storj LabsOr with running\ + \ ..." + title: Use Storj DCS from Cloud-native Environments Using the Sidecar Pattern +title: Use Storj DCS from Cloud-native Environments Using the Sidecar Pattern +--- +Data stored in Storj Decentralized Cloud Storage can be accessed in multiple ways: + +1. With native “uplink” protocol, which connects directly to the nodes where the data is stored +2. With using S3 compatible REST API, using an S3 gateway: + 1. Either the hosted S3 gateway, operated by Storj Labs + 2. Or with running your own S3 gateways + +The easiest way is using the shared S3 gateway with any S3 compatible tool (2.2.) but this approach may also have disadvantages: +1. The encryption keys are shared with the gateway +2. All traffic is routed to the gateway before accessing the data on storage nodes +In a powerful server environment (with enough network and CPU bandwidth) it can be more reasonable to use the native protocol and access the storage nodes directly. However, native protocol is not supported by as many tools as the S3 protocol. + +Fortunately, in Kubernetes – thanks to the sidecar pattern – using the native protocol is almost as easy as using the shared gateway. +## Sidecar Pattern +The smallest deployable unit in Kubernetes is a pod. Pod is the definition of one or more containers with the attached volumes/resources/network usage. Typically, only one container is included in one pod, but sidecar patterns deploys an additional helper container to each pod. + +As the network namespace is shared inside the pod the main container can access the features of the sidecar container. +![](./6de1c07a519802ed.png)To follow this pattern, we should deploy a sidecar container to each of our application pods. + +Instead of using the hosted, multi-tenant version of the S3 gateway: + +![](./06a4d2d7af413c6a.png) + +We will start a single-tenant S3 gateway with each of the services: + +![](./943dcd25f919a98b.png) +## Getting Started +Let’s start with a simple example: we will create a Jupyter notebook which reads data from a Storj bucket for following data science calculations. + + +A Jupyter notebook can be deployed with the following simplified deployment: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: jupyter +spec: + replicas: 1 + selector: + matchLabels: + app: jupyter + template: + metadata: + labels: + app: jupyter + spec: + containers: + - name: jupyter + image: jupyter/base-notebook + ports: + - containerPort: 8888 + hostPort: 8888 +``` + +(Please note that we use **hostPort** here. In a real cluster, service (load balanced, nodeIp) or ingress definition would be required, depending on the environment.) + + +After deploying this definition to a Kubernetes cluster, we can access the Jupyter notebook application and write our own notebooks. + +To open the Jupyter web application we need the secret token which is printed out to the standard output of the container: + +``` +kubectl logs -l app=jupyter -c jupyter --tail=-1 + +.... +[I 14:12:50.361 NotebookApp] Serving notebooks from local directory: /home/jovyan +[I 14:12:50.361 NotebookApp] Jupyter Notebook 6.4.6 is running at: +[I 14:12:50.362 NotebookApp] http://jupyter-7546dc9f8c-ww4hb:8888/?token=32bc4f4617fcad6001895c966ce8df539f5f71a243197d5d +[I 14:12:50.362 NotebookApp] or http://127.0.0.1:8888/?token=32bc4f4617fcad6001895c966ce8df539f5f71a243197d5d +[I 14:12:50.362 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). +[C 14:12:50.366 NotebookApp] + + To access the notebook, open this file in a browser: + file:///home/jovyan/.local/share/jupyter/runtime/nbserver-8-open.html + Or copy and paste one of these URLs: + http://jupyter-7546dc9f8c-ww4hb:8888/?token=32bc4f4617fcad6001895c966ce8df539f5f71a243197d5d + or http://127.0.0.1:8888/?token=32bc4f4617fcad6001895c966ce8df539f5f71a243197d5d +``` + +And now we can create a new notebook where we use the Storj data: + +![](./4b79f2e770056a31.png) + +To read data via S3 protocol, we need *boto*, the python S3 library, which can be added to the docker image or installed as a first step in the notebook: + +```python +import subprocess +subprocess.run(["pip", "install","boto3", "pandas"]) +``` + +Next, we can read/use files directly from Storj: + +```python +import boto3 +import pandas as pd +from io import StringIO + +session = boto3.session.Session() +s3_client = session.client( + 's3', + aws_access_key_id="...", + aws_secret_access_key="...", + endpoint_url="https://gateway.eu1.storjshare.io") + +response = client.get_object(Bucket="sidecar", Key="data.csv") +csv = response["Body"].read().decode('utf-8') +df = pd.read_csv(StringIO(csv)) + +df +``` + +The approach uses the shared S3 gateway and requires access key and secret credentials generated as documented [here](docId:yYCzPT8HHcbEZZMvfoCFa). + +## Activating the Sidecar +Let’s improve the previous example by using the sidecar pattern. First, we need to generate an [*access grant*](docId:b4-QgUOxVHDHSIWpAf3hG) instead of the S3 credentials to access Storj data, and we should define any S3 credentials for our local, single-tenant S3 gateway: + + +![](./c911680feffbe653.png) + +Let’s create a Kubernetes secret with all of these: + +```bash +export ACCESS_GRANT=...generated_by_ui… + +kubectl create secret generic storj-gateway \ +--from-literal=storj-gateway-key=$(pwgen -n 18) \ +--from-literal=storj-gateway-secret=$(pwgen -n 18) \ +--from-literal=storj-access-grant=$ACCESS_GRANT +``` + +Now we can enhance our Kubernetes deployment by adding one more container (put it under spec/template/spec/containers): + +```yaml + - name: storj-sidecar + image: storjlabs/gateway + args: + - run + env: + - name: STORJ_MINIO_ACCESS_KEY + valueFrom: + secretKeyRef: + name: storj-gateway + key: storj-gateway-key + - name: STORJ_MINIO_SECRET_KEY + valueFrom: + secretKeyRef: + name: storj-gateway + key: storj-gateway-secret + - name: STORJ_ACCESS + valueFrom: + secretKeyRef: + name: storj-gateway + key: storj-access-grant +``` + +This container is configured to access the Storj API (using the STORJ\_ACCESS environment variable) and secured by STORJ\_MINIO\_ACCESS\_KEY and STORJ\_MINIO\_SECRET\_KEY. + +Now we can access any Storj object from our Storj bucket, but we can also make it more secure without hard-coding credentials during the initialization of the python S3 client. We should add two more environment variables to the existing Jupyter container to make it available for the client: + +```yaml + spec: + containers: + - name: jupyter + image: jupyter/base-notebook + ports: + - containerPort: 8888 + hostPort: 8888 + env: + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: storj-gateway + key: storj-gateway-key + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: storj-gateway + key: storj-gateway-secret + - name: storj-sidecar +``` + +With this approach, we can initialize the S3 client without hard-coding the credentials: + +```python +session = boto3.session.Session() +client = session.client( + 's3', + endpoint_url="http://localhost:7777") +``` + +Note: http://localhost:7777 is the address of the single-tenant Storj gateway which is running in the same pod. + +## Summary +Sidecar pattern is an easy way to access our data from Storj Decentralized Cloud Storage using the power of the native protocol, even if our application is compatible only with the S3 Rest API. +![](./e76c6da3bc2247e7.png) + + diff --git a/app/(blog)/blog/using-storj-dcs-with-github-actions/1da95eeb5ef706b6.png b/app/(blog)/blog/using-storj-dcs-with-github-actions/1da95eeb5ef706b6.png new file mode 100644 index 000000000..fe05553bf Binary files /dev/null and b/app/(blog)/blog/using-storj-dcs-with-github-actions/1da95eeb5ef706b6.png differ diff --git a/app/(blog)/blog/using-storj-dcs-with-github-actions/334785fee7e124be.png b/app/(blog)/blog/using-storj-dcs-with-github-actions/334785fee7e124be.png new file mode 100644 index 000000000..725b0578f Binary files /dev/null and b/app/(blog)/blog/using-storj-dcs-with-github-actions/334785fee7e124be.png differ diff --git a/app/(blog)/blog/using-storj-dcs-with-github-actions/87361ca4a0a843f4.png b/app/(blog)/blog/using-storj-dcs-with-github-actions/87361ca4a0a843f4.png new file mode 100644 index 000000000..29fffaa4f Binary files /dev/null and b/app/(blog)/blog/using-storj-dcs-with-github-actions/87361ca4a0a843f4.png differ diff --git a/app/(blog)/blog/using-storj-dcs-with-github-actions/9ccdc75d22b6993e.png b/app/(blog)/blog/using-storj-dcs-with-github-actions/9ccdc75d22b6993e.png new file mode 100644 index 000000000..7a79cb44f Binary files /dev/null and b/app/(blog)/blog/using-storj-dcs-with-github-actions/9ccdc75d22b6993e.png differ diff --git a/app/(blog)/blog/using-storj-dcs-with-github-actions/9fae43e73a66cba0.png b/app/(blog)/blog/using-storj-dcs-with-github-actions/9fae43e73a66cba0.png new file mode 100644 index 000000000..ca7f05502 Binary files /dev/null and b/app/(blog)/blog/using-storj-dcs-with-github-actions/9fae43e73a66cba0.png differ diff --git a/app/(blog)/blog/using-storj-dcs-with-github-actions/page.md b/app/(blog)/blog/using-storj-dcs-with-github-actions/page.md new file mode 100644 index 000000000..1be132734 --- /dev/null +++ b/app/(blog)/blog/using-storj-dcs-with-github-actions/page.md @@ -0,0 +1,169 @@ +--- +author: + name: Kaloyan Raev +date: '2021-08-31 00:00:00' +heroimage: ./9ccdc75d22b6993e.png +layout: blog +metadata: + description: GitHub Actions is their system to automate, customize, and execute + software development workflows in the GitHub repository. This article will inform + you how to upload files to a Storj DCS bucket from a GitHub Actions workflow.The + Storj DCS Public Network Stats is one of the projects at Storj wher... + title: Using Storj DCS with GitHub Actions +title: Using Storj DCS with GitHub Actions + +--- + +[GitHub Actions](https://docs.github.com/en/actions) is their system to automate, customize, and execute software development workflows in the GitHub repository. This article will inform you how to upload files to a Storj DCS bucket from a GitHub Actions workflow. + + +The [Storj DCS Public Network Stats](https://stats.storjshare.io/) is one of the projects at Storj where we use GitHub Actions. The statistics are hosted as a [static website](docId:GkgE6Egi02wRZtyryFyPz) on Storj DCS, so we have an easy way to redeploy the homepage when we merge any modification in the code repository. We created a GitHub Actions workflow that converts the Markdown file of the homepage to an HTML file and then uploads it to the bucket hosting the website. + + +GitHub Actions has a marketplace for actions created by the community. Instead of creating our own Storj-specific action to upload files to Storj DCS, we decided to keep it simple and use the [s3-sync-action](https://github.com/jakejarvis/s3-sync-action) that the community has already created. The s3-sync-action allows uploading files to an S3-compatible storage service, so we took advantage of Storj Gateway-MT - the globally available, multi-region hosted S3-compatible gateway. + + +Let’s break down the specific GitHub Actions workflow for the Storj DCS Public Network Stats project. [The complete workflow is here](https://github.com/storj/stats/blob/main/.github/workflows/upload-homepage.yml). + + +Every workflow starts with declaring its name: + + +``` + +# This is a workflow converts homepage.md to index.html +# and uploads it to the static website +name: upload homepage + +``` + + + + +Then follow the rules for triggering the workflow: + + + +``` + +# Controls when the workflow will run +on: + # Triggers the workflow only on push event to the main branch, + # but not for pull requests + push: + branches: [ main ] + # Triggers the workflow only if the homepage.md file has been # edited + paths: + - 'homepage.md' + +``` + + + + +In this case, the workflow triggers when a commit is merged to the main branch, and that commit modifies the homepage.md file. + + +Next, we have the definition of the job that will be run when the above event triggers: + + + +``` + +# A workflow run is made up of one or more jobs that can run +# sequentially or in parallel +jobs: + # This workflow contains a single job called "build" + build: + # The type of runner that the job will run on + runs-on: ubuntu-latest + # Steps represent a sequence of tasks that will be executed as +# part of the job + steps: + +``` + + + + +The job will run on an Ubuntu VM and will execute the following three steps: + +1. Check out the head of the GitHub repository + +``` + +# Checks-out your repository under $GITHUB_WORKSPACE, +# so your job can access it +- uses: actions/checkout@v2 + +``` +2. Convert the homepage.md file to index.html + +``` + +# Converts the homepage.md file to index.html +- uses: ZacJW/markdown-html-action@1.1.0 + with: + input_files: '[["homepage.md"]]' + output_files: '["index.html"]' + extensions: '[]' # Alas, this cannot be skipped even if empty + +``` +3. Upload the index.html file to the Storj DCS bucket. + +``` + +# Uploads the index.html file to the root of the destination bucket +- uses: jakejarvis/s3-sync-action@v0.5.1 + with: + # This is a workaround as SOURCE_DIR does not support + # a single file + args: --exclude '*' --include 'index.html' + env: + AWS_S3_ENDPOINT: ${{ secrets.AWS_S3_ENDPOINT }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + +``` + +The destination bucket and the S3 credentials for s3-sync-action are configured through environment variables. In this case, we use [encrypted secrets](https://docs.github.com/en/actions/reference/encrypted-secrets) from the GitHub repository to keep this information private and safe from public eyes. + + +Encrypted secrets can be configured in the “Secrets” section of the repository settings.  + + +![](./9fae43e73a66cba0.png) + +All these secrets can be created via the Storj Satellite web interface. After logging in to the web interface, we make sure that the target bucket is already created. If not, the easiest way to create it is using the [Object Browser](docId:4oDAezF-FcfPr0WPl7knd). Then we set the name of the bucket as the AWS\_S3\_BUCKET secret in the Github repository. + + +Having the bucket created, next, we create S3 credentials that grant access to that bucket. This is done by [creating a new access grant from the web interface](docId:AsyYcUJFbO1JI8-Tu8tW3#generate-s3-compatible-credentials). + + +In the Permissions dialog, we make sure to limit the access only to the target bucket instead of giving access to the whole project. + +![](./87361ca4a0a843f4.png) + +In the Access Grant dialog, we click on the Generate S3 Gateway Credentials button. + + +![](./334785fee7e124be.png) + +This generates the S3 credentials for the access grant that can be used with Gateway-MT. + + +![](./1da95eeb5ef706b6.png) + +We use these credentials to set the remaining secrets in the Github repository: + +* AWS\_ACCESS\_KEY\_ID is set to the Access Key value +* AWS\_SECRET\_ACCESS\_KEY is set to the Secret Key value +* AWS\_S3\_ENDPOINT is set to the End Point value + +With this, everything is now complete to run the GitHub Actions workflow successfully. + + +If you have any questions, please feel free to reach out to us at [support@storj.io](mailto:support@storj.io) or visit .  + + diff --git a/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/29e62398803ba277.png b/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/29e62398803ba277.png new file mode 100644 index 000000000..57b959424 Binary files /dev/null and b/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/29e62398803ba277.png differ diff --git a/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/47f263c59b58c3b7.jpeg b/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/47f263c59b58c3b7.jpeg new file mode 100644 index 000000000..f359ee0ef Binary files /dev/null and b/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/47f263c59b58c3b7.jpeg differ diff --git a/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/6371747a59800613.jpeg b/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/6371747a59800613.jpeg new file mode 100644 index 000000000..804086caa Binary files /dev/null and b/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/6371747a59800613.jpeg differ diff --git a/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/page.md b/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/page.md new file mode 100644 index 000000000..2291d8fd3 --- /dev/null +++ b/app/(blog)/blog/visualizing-decentralized-data-distribution-with-the-linkshare-object-map/page.md @@ -0,0 +1,50 @@ +--- +author: + name: Brandon Iglesias +date: '2020-10-21 00:00:00' +heroimage: ./6371747a59800613.jpeg +layout: blog +metadata: + description: At Storj Labs we're distributed system junkies. We enjoy building highly + distributed, ridiculously resilient software. The Storj Network is currently spread + across over 10,000 uncorrelated endpoints, and that number is growing fast.The + global substrate of diverse, uncorrelated endpoints across wh... + title: Visualizing Decentralized Data Distribution with the Linksharing Object Map +title: Visualizing Decentralized Data Distribution with the Linksharing Object Map + +--- + +At Storj Labs we're distributed system junkies. We enjoy building highly distributed, ridiculously resilient software. The Storj Network is currently spread across over 10,000 uncorrelated endpoints, and that number is growing fast. + +The global substrate of diverse, uncorrelated endpoints across which the network runs is unmatched by any other cloud provider. + +Storage Nodes run across a diverse distribution of operating systems, hardware types, geographic locations, and owners. Node Operator software runs on Linux systems like Ubuntu, CentOS, Debian, and Fedora, as well as macOS and Windows, with a native MSI installer. Storage Nodes run in Docker containers as well as compile to native binaries for ARM and AMD64. Hardware ranges from basic Raspberry Pis to QNAP NAS devices. + +Now that we have Storage Nodes across the world, we decided to build a simple visualization tool to showcase just how distributed and resilient the files stored on Tardigrade actually are. + +### Distributed Storage, Visualized + +Through this tool—which is called the Linksharing Object Map—our team and our community can visualize the geographic distribution of data uploaded to our Tardigrade service. This showcases how resilient the network is, as well as the wide geographic distribution of Nodes holding each object. + +We set out to build the Linksharing Object Map Dashboard at the start of the two-day Storj Labs employee hackathon and quickly productized and completed the project. + +![](./29e62398803ba277.png)![](./47f263c59b58c3b7.jpeg)Try it out yourself by generating access for an object, and creating a link share for the URL, [like outlined in our documentation](https://documentation.tardigrade.io/getting-started/uploading-your-first-object/view-distribution-of-an-object). This process will generate a link with a macaroon (embedded, [hash-based logic](https://storj.io/blog/2019/12/secure-access-control-in-the-decentralized-cloud/)) that controls how the object can be accessed. + +See an example of the Node map yourself, here: [Link share Object Map](https://bit.ly/31qVdyc) + +### Uplink Visualizer: The tech and how it works + +The Uplink Visualizer is a simple GoLang application that ingests, transforms, and visualizes client-side data from the uplink client. The tool grabs the IP addresses of the Storage Nodes holding pieces for a given object and displays them on a map. + +Essentially, there's an endpoint on the Satellite that will return the IP address of all the Storage Nodes holding pieces for an object on the network if you have permission to download it with your API key. + +We use [MaxMind](https://github.com/maxmind/) to convert the list of the IP addresses to their corresponding global longitudes and latitudes. We then use [LeafletJS](https://leafletjs.com/) to message the geo locations to be displayed on the leaflet js map. + +### Try it yourself, and be the cloud! + +We're excited for you to try the Uplink Object Map and check out its [code](https://github.com/storj/linksharing). It's licensed under Apache-2.0 License, and because it's open source (like most of the code we produce), you can also contribute to the tool as well. If you have any feedback on the visualizer or find it useful, please let us know at . + +Finally, if you like the look and feel of the tool, please let the world know, and tweet it to us @storjproject—we always reshare and promote our community's content and efforts! + +Please note that this is a GENERIC location so the Storage Nodes actual location is not disclosed. A user is ONLY able to get this information for a file if they have permission to download it. + diff --git a/app/(blog)/layout.js b/app/(blog)/layout.js new file mode 100644 index 000000000..fbede7f4f --- /dev/null +++ b/app/(blog)/layout.js @@ -0,0 +1,18 @@ +import '@/styles/tailwind.css' + +export const metadata = { + title: { + template: '%s | Storj Engineering Blog', + default: 'Storj Engineering Blog', + }, + description: + 'Learn about the latest developments in the Storj network and the technology that powers it.', +} + +export default function RootLayout({ + // Layouts must accept a children prop. + // This will be populated with nested layouts or pages + children, +}) { + return children +} diff --git a/app/dcs/access/page.md b/app/(docs)/dcs/access/page.md similarity index 73% rename from app/dcs/access/page.md rename to app/(docs)/dcs/access/page.md index 096bff475..8474348d7 100644 --- a/app/dcs/access/page.md +++ b/app/(docs)/dcs/access/page.md @@ -31,21 +31,15 @@ The Access Grant screen allows you to create or delete Access Grants, generate c ## Create Keys for CLI -1. You need to have a satellite account and Uplink CLI installed. See [](docId:HeEf9wiMdlQx9ZdS_-oZS) +1. You need to have a Storj account and Uplink CLI installed. See [](docId:HeEf9wiMdlQx9ZdS_-oZS) 2. To start, proceed through the initial steps of creating a new Access Grant. -3. Navigate to "Access" page and click the **Create Keys for CLI** link (rightmost option). +3. Navigate to "Access Keys" page and click the **New Access Key** button, then type an access name and choose **API Key** as an Access type. - ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/dBMRUSzPsBClfxJaXwk7S_cli-keys.png) +4. On the next step, select either **Full Access** or **Advanced** if you want to choose the permissions, buckets, and set an expiry date for this access key. -4. Provide name, permissions and optionally buckets, select **Create Keys**. - - ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/NZGAlqeSEv-vuzJW1enUW_cli-keys2.png) - -5. Copy and save the **Satellite Address** and **API Key **in a safe place or download them as they will only appear once. - - ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/68ftNpbXKmJkroQyO2C9Q_cli-keys3.png) +5. Once you create the access key, copy and save the **Satellite Address** and **API Key** in a safe place, or download them as they will only appear once. 6. Make sure you've already [](docId:hFL-goCWqrQMJPcTN82NB) and run `uplink setup`. @@ -80,15 +74,15 @@ The Access Grant screen allows you to create or delete Access Grants, generate c {% code-group %} ```windows - ./uplink.exe share --readonly=false + ./uplink.exe access restrict --readonly=false ``` ```linux - uplink share --readonly=false + uplink access restrict --readonly=false ``` ```macos - uplink share --readonly=false + uplink access restrict --readonly=false ``` {% /code-group %} @@ -100,7 +94,7 @@ The Access Grant screen allows you to create or delete Access Grants, generate c 9. Your Access Grant should have been output. {% callout type="success" %} -The alternative for using the uplink setup command and then uplink share is to use the `uplink access create` command instead, it will print the Access Grant right away. +The alternative for using the `uplink setup` command and then `uplink access restrict` is to use the `uplink access create` command instead, it will print the Access Grant right away. {% /callout %} --- @@ -109,12 +103,6 @@ The alternative for using the uplink setup command and then uplink share is to u To Delete an Access Grant, select three dots on the right side of the Access Grant and choose **Delete Access**: -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/8DjOwU27KCkISKOJs9T3O_access10.png) - -Then confirm that you want to delete the Access Grant by typing its name and confirming with **Delete Access** button. - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/--lULF4MsGMwhbtfyIa5W_access11.png) - {% callout type="danger" %} **Important:** If you delete an Access Grant from the Satellite user interface, that Access Grant will immediately cease to function, and all hierarchically derived child Access Grants and Storj gateway access credentials based on that Access Grant will also cease to function. Any data uploaded with that Access Grant will persist on Storj. If you didn't back up the Encryption Passphrase used with the Access Grant you are deleting, you will not be able to decrypt that data without that Encryption Passphrase, and it will be effectively unrecoverable. {% /callout %} diff --git a/app/dcs/api/_meta.json b/app/(docs)/dcs/api/_meta.json similarity index 100% rename from app/dcs/api/_meta.json rename to app/(docs)/dcs/api/_meta.json diff --git a/app/dcs/api/s3/credentials/page.md b/app/(docs)/dcs/api/s3/credentials/page.md similarity index 100% rename from app/dcs/api/s3/credentials/page.md rename to app/(docs)/dcs/api/s3/credentials/page.md diff --git a/app/dcs/api/s3/multipart-upload/multipart-part-size/page.md b/app/(docs)/dcs/api/s3/multipart-upload/multipart-part-size/page.md similarity index 100% rename from app/dcs/api/s3/multipart-upload/multipart-part-size/page.md rename to app/(docs)/dcs/api/s3/multipart-upload/multipart-part-size/page.md diff --git a/app/dcs/api/s3/multipart-upload/page.md b/app/(docs)/dcs/api/s3/multipart-upload/page.md similarity index 100% rename from app/dcs/api/s3/multipart-upload/page.md rename to app/(docs)/dcs/api/s3/multipart-upload/page.md diff --git a/app/(docs)/dcs/api/s3/object-lock/page.md b/app/(docs)/dcs/api/s3/object-lock/page.md new file mode 100644 index 000000000..33539751a --- /dev/null +++ b/app/(docs)/dcs/api/s3/object-lock/page.md @@ -0,0 +1,147 @@ +--- +title: Object Lock (Beta) +docId: gjrGzPNnhpYrAGTTAUaj +metadata: + description: Detailed guide on the Beta of object lock + title: 'Object Lock (Beta)' +--- +{% callout type="info" %} +**Beta Service Release** + +S3-Compatible Object Lock is currently available as pre-production beta and may not be suitable for all production environments. + +Please refer to our [Terms of Service](https://www.storj.io/legal/terms-of-service) for more information on release definitions. +{% /callout %} + +## Summary + +### Overview +Storj is currently developing [](docId:oogh5vaiGei6atohm5thi). As part of that development, we have released an early Beta Service Release, which includes **S3-Compatible Object Lock**, that can be used to evaluate the functionality. + +## Beta Opt-In Instructions +To gain access to S3-compatible Object Lock, you'll need to opt into the **Object Versioning Beta** following the steps below: +{% partial file="versioning-opt-in.md" /%} + +## Enabling Object Lock on a Bucket +{% callout type="info" %} +Object Lock is not backwards compatible with existing buckets—a new bucket must be created to enable Object Lock. + +Additionally, PutObjectLockConfiguration is not supported yet, so Object Lock can only be enabled during bucket creation and cannot be added afterward. +{% /callout %} + +{% partial file="create-bucket.md" /%} + +### FAQ +- **When will Object Lock be fully released?** + - We are actively working on Governance Mode and Legal Hold and plan to add them to the beta by as soon as they are ready. + - The full feature set will be released as generally available some time after the beta is feature complete +- **Why would I test this Beta?** + - If you plan to conduct a proof of concept or want to test a a pre-production instance of an integration that utilizes object lock. +- **How do I test the Beta?** + - See instructions above +- **Why doesn't the initial version support Governance Mode and Legal Hold?** + - Our initial goal was to implement the most common S3 features regarding immutability and deliver them as quickly as possible, which led us to focus on the strictest form of object lock: Compliance Mode. + - We are actively working on Governance Mode, Legal Hold, and the `PutObjectLockConfiguration` action and will add them to the beta as soon as they are ready +- **How do I give feedback or request features related to Object Lock?** + - Our roadmap is public. Here are the relevant roadmap items: + - [S3 Object Lock: Compliance Mode](https://github.com/storj/roadmap/issues/47) + - [S3 Object Lock: Add Governance Mode and Legal Hold](https://github.com/storj/roadmap/issues/98) + - You may also submit a support request here: [https://support.storj.io/hc/en-us/requests/new](https://support.storj.io/hc/en-us/requests/new) + +## Technical Details +### New S3 Actions Supported: +{% table %} +* Action +* API Description +* Description of Change(s) +--- +* GetObjectLockConfiguration +* Gets the object lock configuration for a bucket. +* Will return the ObjectLockConfiguration with `ObjectLockEnabled` either as `Enabled` or empty. + + `Rule` will not be included as a response element as specifying a bucket-level object Lock rule is initially out of scope. +--- +* PutObjectRetention +* Places an object retention configuration on an object. +* The only value supported for `Mode` is `COMPLIANCE` as Governance Mode is initially out of scope. +--- +* GetObjectRetention +* Retrieves an object's retention settings. +* +{% /table %} + +### Existing S3 Actions Updated +{% table %} +* Action +* API Description +* Description of Change(s) +--- +* CreateBucket +* Creates a new bucket. +* CreateBucket will now accept the following request parameter: + * `x-amz-bucket-object-lock-enabled` +--- +* HeadObject +* Retrieves metadata from an object without returning the object itself. +* HeadObject will now return: + * Mode (only Compliance is supported initially) that is currently in place for the requested object + * Date/time that the object's lock will expire +--- +* GetObject +* Retrieves an object from a bucket. +* GetObject will now return: + * Mode (only Compliance is supported initially) that is currently in place for the requested object + * Date/time that the object's lock will expire +--- +* PutObject +* Adds an object to a bucket. +* PutObject will now: + * Prevent locked object versions from being overwritten + + PutObject will now accept the following request parameters: + * `x-amz-object-lock-mode` (only Compliance is supported initially) + * `x-amz-object-lock-retain-until-date` +--- +* CopyObject +* Creates a copy of an object that is already stored on Storj. +* CopyObject will now accept the following request parameters: + * `x-amz-object-lock-mode` (only Compliance is supported initially) + * `x-amz-object-lock-retain-until-date` +--- +* CreateMultipartUpload +* This action initiates a multipart upload and returns an upload ID. +* CreateMultipartUpload will now accept the following request parameters: + * `x-amz-object-lock-mode` (only Compliance is supported initially) + * `x-amz-object-lock-retain-until-date` + + Storj has a unique object level TTL. Any request that has both a TTL and a retention period will be rejected to prevent TTL's from conflicting with object lock retention periods. +--- +* DeleteBucket +* Deletes the specified bucket. +* Forced deletion of a bucket with locked objects will be prevented. +--- +* DeleteObject +* Removes an object from a bucket. +* Deletion of an object with a retention set will be prevented. +{% /table %} + +### Actions not yet available (currently in active development) +{% table %} +* Action +* API Description +* Description of Change(s) +--- +* PutObjectLockConfiguration +* Enables Object Lock configuration on a bucket. +* **ObjectLockEnabled**: Indicates if Object Lock is enabled on the bucket. + + **Rule**: Specifies the Object Lock rule (mode and period) for the bucket. The period can be either `Days` or `Years`. +--- +* GetObjectLegalHold +* Retrieves the Legal Hold status of an object. +* +--- +* PutObjectLegalHold +* Applies a Legal Hold to the specified object. +* +{% /table %} diff --git a/app/dcs/buckets/object-versioning/page.md b/app/(docs)/dcs/api/s3/object-versioning/page.md similarity index 60% rename from app/dcs/buckets/object-versioning/page.md rename to app/(docs)/dcs/api/s3/object-versioning/page.md index 2d6fca136..acaf9edc6 100644 --- a/app/dcs/buckets/object-versioning/page.md +++ b/app/(docs)/dcs/api/s3/object-versioning/page.md @@ -4,21 +4,24 @@ docId: oogh5vaiGei6atohm5thi metadata: description: Detailed guide on enabling object versioning for buckets title: 'Object Versioning in Storj' +redirects: + - /dcs/buckets/object-versioning --- -Storj has released S3 Compatible Object Versioning in open beta. +{% callout type="info" %} +**Beta Service Release** -## How do I opt in to the object versioning beta? +S3-Compatible Object Versioning is currently available as pre-production beta and may not be suitable for all production environments. + +Please refer to our [Terms of Service](https://www.storj.io/legal/terms-of-service) for more information on release definitions. +{% /callout %} -You need to opt in to the object versioning beta per project. To opt in to the object versioning beta, you can follow these steps: -1. Login to the Storj web console -2. Navigate to the desired project -3. You will be prompted to enable object versioning for the project +## How do I opt in to the object versioning beta? -{% callout type="warning" %} -Object versioning is currently in open beta and may not be suitable for all production environments. -{% /callout %} +To opt in to the object versioning beta, you can follow these steps: + +{% partial file="versioning-opt-in.md" /%} ## How does object versioning work? @@ -32,7 +35,7 @@ Object versioning enables you to preserve, retrieve, and restore every version o - **Recovery and Rollback:** In case of accidental deletion or if an object is overwritten with an undesired version, you can easily recover the previous version of the object. - +- **Object Lock:** For more details on Object Lock support see [](docId:gjrGzPNnhpYrAGTTAUaj). {% callout type="info" %} Note that enabling object versioning can increase storage costs since each version of an object is stored separately. @@ -40,7 +43,7 @@ Note that enabling object versioning can increase storage costs since each versi ### Supported S3 API Methods for Object Versioning -Storj's S3 Compatible Object Versioning supports a range of S3 API methods, allowing you to manage and interact with versioned objects. Below are the key S3 API methods supported by Storj's object versioning, along with a brief description of their use: +Storj's S3-Compatible Object Versioning supports a range of S3 API methods, allowing you to manage and interact with versioned objects. Below are the key S3 API methods supported by Storj's object versioning, along with a brief description of their use: #### Bucket Operations @@ -52,4 +55,15 @@ Storj's S3 Compatible Object Versioning supports a range of S3 API methods, allo - **PUT Object**: Adds an object to a bucket. If versioning is enabled, a unique version ID is assigned to the object. - **GET Object**: Retrieves the current version of an object or a specific version if the version ID is specified. - **DELETE Object**: Permanently deletes a version of an object if the version ID is provided, or marks the current version as deleted by adding a delete marker. -- **LIST Versions**: Lists all the versions of all objects in a bucket, including delete markers. \ No newline at end of file +- **LIST Versions**: Lists all the versions of all objects in a bucket, including delete markers. + +### Bucket Versioning Status + +The following are the possible versioning statuses a bucket can be in: + +| Status | Description | +| ------------ | ----------------------------------------------------------------------------------------------- | +| Not Supported| The bucket was created prior to the release of object versioning, and versioning cannot be enabled. Create a new bucket to enable versioning. | +| Unversioned | Versioning has not been set on the bucket. | +| Enabled | Versioning is enabled for the bucket. | +| Suspended | Versioning was previously enabled, but is currently suspended. You may re-enable versioning at any time. | \ No newline at end of file diff --git a/app/dcs/api/s3/presigned-urls/page.md b/app/(docs)/dcs/api/s3/presigned-urls/page.md similarity index 100% rename from app/dcs/api/s3/presigned-urls/page.md rename to app/(docs)/dcs/api/s3/presigned-urls/page.md diff --git a/app/dcs/api/s3/s3-compatibility/page.md b/app/(docs)/dcs/api/s3/s3-compatibility/page.md similarity index 93% rename from app/dcs/api/s3/s3-compatibility/page.md rename to app/(docs)/dcs/api/s3/s3-compatibility/page.md index 97fe9078e..3feaa1b29 100644 --- a/app/dcs/api/s3/s3-compatibility/page.md +++ b/app/(docs)/dcs/api/s3/s3-compatibility/page.md @@ -45,7 +45,7 @@ The Storj S3-compatible Gateway supports a RESTful API that is compatible with t | GetBucketLifecycle (deprecated) | No | | | GetBucketLifecycleConfiguration | No | | | GetBucketLocation | Full | See GetBucketLocation section | -| GetBucketLogging | No | | +| GetBucketLogging | No | Available upon request; see Bucket Logging section below | | GetBucketMetricsConfiguration | No | | | GetBucketNotification (deprecated) | No | | | GetBucketNotificationConfiguration | No | | @@ -59,9 +59,9 @@ The Storj S3-compatible Gateway supports a RESTful API that is compatible with t | GetBucketWebsite | No | | | GetObject | Partial | We need to add support for the partNumber parameter | | GetObjectAcl | No | | -| GetObjectLegalHold | No | | -| GetObjectLockConfiguration | No | | -| GetObjectRetention | No | | +| GetObjectLegalHold | No | Currently in active development. See [](docId:gjrGzPNnhpYrAGTTAUaj) | +| GetObjectLockConfiguration | No | Currently in active development. See [](docId:gjrGzPNnhpYrAGTTAUaj) | +| GetObjectRetention | Yes (Beta) | [](docId:gjrGzPNnhpYrAGTTAUaj) | | GetObjectTagging | Full | Tags can be modified outside of tagging endpoints | | GetObjectTorrent | No | | | GetPublicAccessBlock | No | | @@ -73,7 +73,7 @@ The Storj S3-compatible Gateway supports a RESTful API that is compatible with t | ListBucketMetricsConfigurations | No | | | ListBuckets | Full | | | ListMultipartUploads | Partial | See ListMultipartUploads section | -| ListObjectVersions | Yes (Beta) | | +| ListObjectVersions | Yes (Beta) | [](docId:gjrGzPNnhpYrAGTTAUaj) | | ListObjects | Partial | See ListObjects section | | ListObjectsV2 | Partial | See ListObjects section | | ListParts | Full | | @@ -86,7 +86,7 @@ The Storj S3-compatible Gateway supports a RESTful API that is compatible with t | PutBucketInventoryConfiguration | No | | | PutBucketLifecycle (deprecated) | No | | | PutBucketLifecycleConfiguration | No | | -| PutBucketLogging | No | | +| PutBucketLogging | No | Available upon request; see Bucket Logging section below | | PutBucketMetricsConfiguration | No | | | PutBucketNotification (deprecated) | No | | | PutBucketNotificationConfiguration | No | | @@ -95,13 +95,13 @@ The Storj S3-compatible Gateway supports a RESTful API that is compatible with t | PutBucketReplication | No | | | PutBucketRequestPayment | No | Planned support status needs verification | | PutBucketTagging | No | | -| PutBucketVersioning | Yes (Beta) | | +| PutBucketVersioning | Yes (Beta) | [](docId:gjrGzPNnhpYrAGTTAUaj) | | PutBucketWebsite | No | | | PutObject | Full | | | PutObjectAcl | No | | | PutObjectLegalHold | No | | -| PutObjectLockConfiguration | No | | -| PutObjectRetention | No | | +| PutObjectLockConfiguration | No | Currently in active development. See [](docId:gjrGzPNnhpYrAGTTAUaj) | +| PutObjectRetention | Yes (Beta) | [](docId:gjrGzPNnhpYrAGTTAUaj) | | PutObjectTagging | Full | Tags can be modified outside of tagging endpoints | | PutPublicAccessBlock | No | | | RestoreObject | No | | @@ -120,6 +120,9 @@ Full compatibility means that we support all features of a specific action except for features that rely on other actions that we haven't fully implemented. +### Bucket Logging +Bucket Logging is available upon request. Please refer to [](docId:0191fc71-e031-761c-a16b-aa8ca9e44413). + ### GetBucketLocation This is currently supported in Gateway-MT only. @@ -341,4 +344,4 @@ func main() { fmt.Println(awsutil.Prettify(output)) } -``` +``` \ No newline at end of file diff --git a/app/dcs/api/s3/s3-compatible-gateway/page.md b/app/(docs)/dcs/api/s3/s3-compatible-gateway/page.md similarity index 100% rename from app/dcs/api/s3/s3-compatible-gateway/page.md rename to app/(docs)/dcs/api/s3/s3-compatible-gateway/page.md diff --git a/app/dcs/api/sdk/page.md b/app/(docs)/dcs/api/sdk/page.md similarity index 96% rename from app/dcs/api/sdk/page.md rename to app/(docs)/dcs/api/sdk/page.md index aa9a30dc7..f736f679d 100644 --- a/app/dcs/api/sdk/page.md +++ b/app/(docs)/dcs/api/sdk/page.md @@ -15,7 +15,7 @@ Storj currently has community contributed bindings for Python, Swift, .Net, PHP, Below are Storj's provided bindings: -- [GO](https://github.com/storj/uplink) +- [Go](https://github.com/storj/uplink) - [C](https://github.com/storj/uplink-c) diff --git a/app/dcs/api/storj-ipfs-pinning/page.md b/app/(docs)/dcs/api/storj-ipfs-pinning/page.md similarity index 100% rename from app/dcs/api/storj-ipfs-pinning/page.md rename to app/(docs)/dcs/api/storj-ipfs-pinning/page.md diff --git a/app/dcs/api/uplink-cli/access-command/access-create/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/access-create/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/access-create/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/access-create/page.md diff --git a/app/dcs/api/uplink-cli/access-command/access-export/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/access-export/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/access-export/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/access-export/page.md diff --git a/app/dcs/api/uplink-cli/access-command/access-import/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/access-import/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/access-import/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/access-import/page.md diff --git a/app/dcs/api/uplink-cli/access-command/access-inspect-command/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/access-inspect-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/access-inspect-command/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/access-inspect-command/page.md diff --git a/app/dcs/api/uplink-cli/access-command/access-list-command/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/access-list-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/access-list-command/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/access-list-command/page.md diff --git a/app/dcs/api/uplink-cli/access-command/access-register/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/access-register/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/access-register/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/access-register/page.md diff --git a/app/dcs/api/uplink-cli/access-command/access-remove/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/access-remove/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/access-remove/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/access-remove/page.md diff --git a/app/dcs/api/uplink-cli/access-command/access-restrict/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/access-restrict/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/access-restrict/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/access-restrict/page.md diff --git a/app/dcs/api/uplink-cli/access-command/access-revoke/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/access-revoke/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/access-revoke/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/access-revoke/page.md diff --git a/app/dcs/api/uplink-cli/access-command/access-use/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/access-use/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/access-use/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/access-use/page.md diff --git a/app/dcs/api/uplink-cli/access-command/page.md b/app/(docs)/dcs/api/uplink-cli/access-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/access-command/page.md rename to app/(docs)/dcs/api/uplink-cli/access-command/page.md diff --git a/app/dcs/api/uplink-cli/cp-command/page.md b/app/(docs)/dcs/api/uplink-cli/cp-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/cp-command/page.md rename to app/(docs)/dcs/api/uplink-cli/cp-command/page.md diff --git a/app/dcs/api/uplink-cli/import-command/page.md b/app/(docs)/dcs/api/uplink-cli/import-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/import-command/page.md rename to app/(docs)/dcs/api/uplink-cli/import-command/page.md diff --git a/app/dcs/api/uplink-cli/installation/page.md b/app/(docs)/dcs/api/uplink-cli/installation/page.md similarity index 100% rename from app/dcs/api/uplink-cli/installation/page.md rename to app/(docs)/dcs/api/uplink-cli/installation/page.md diff --git a/app/dcs/api/uplink-cli/ls-command/page.md b/app/(docs)/dcs/api/uplink-cli/ls-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/ls-command/page.md rename to app/(docs)/dcs/api/uplink-cli/ls-command/page.md diff --git a/app/dcs/api/uplink-cli/meta-command/meta-get-command/page.md b/app/(docs)/dcs/api/uplink-cli/meta-command/meta-get-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/meta-command/meta-get-command/page.md rename to app/(docs)/dcs/api/uplink-cli/meta-command/meta-get-command/page.md diff --git a/app/dcs/api/uplink-cli/meta-command/page.md b/app/(docs)/dcs/api/uplink-cli/meta-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/meta-command/page.md rename to app/(docs)/dcs/api/uplink-cli/meta-command/page.md diff --git a/app/dcs/api/uplink-cli/mv/page.md b/app/(docs)/dcs/api/uplink-cli/mv/page.md similarity index 100% rename from app/dcs/api/uplink-cli/mv/page.md rename to app/(docs)/dcs/api/uplink-cli/mv/page.md diff --git a/app/dcs/api/uplink-cli/page.md b/app/(docs)/dcs/api/uplink-cli/page.md similarity index 100% rename from app/dcs/api/uplink-cli/page.md rename to app/(docs)/dcs/api/uplink-cli/page.md diff --git a/app/dcs/api/uplink-cli/rb-command/page.md b/app/(docs)/dcs/api/uplink-cli/rb-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/rb-command/page.md rename to app/(docs)/dcs/api/uplink-cli/rb-command/page.md diff --git a/app/dcs/api/uplink-cli/rm-command/page.md b/app/(docs)/dcs/api/uplink-cli/rm-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/rm-command/page.md rename to app/(docs)/dcs/api/uplink-cli/rm-command/page.md diff --git a/app/dcs/api/uplink-cli/setup-command/page.md b/app/(docs)/dcs/api/uplink-cli/setup-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/setup-command/page.md rename to app/(docs)/dcs/api/uplink-cli/setup-command/page.md diff --git a/app/dcs/api/uplink-cli/share-command/page.md b/app/(docs)/dcs/api/uplink-cli/share-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/share-command/page.md rename to app/(docs)/dcs/api/uplink-cli/share-command/page.md diff --git a/app/dcs/api/uplink-cli/uplink-mb-command/page.md b/app/(docs)/dcs/api/uplink-cli/uplink-mb-command/page.md similarity index 100% rename from app/dcs/api/uplink-cli/uplink-mb-command/page.md rename to app/(docs)/dcs/api/uplink-cli/uplink-mb-command/page.md diff --git a/app/dcs/buckets/_meta.json b/app/(docs)/dcs/buckets/_meta.json similarity index 100% rename from app/dcs/buckets/_meta.json rename to app/(docs)/dcs/buckets/_meta.json diff --git a/app/(docs)/dcs/buckets/bucket-logging/page.md b/app/(docs)/dcs/buckets/bucket-logging/page.md new file mode 100644 index 000000000..11f0f6a6c --- /dev/null +++ b/app/(docs)/dcs/buckets/bucket-logging/page.md @@ -0,0 +1,117 @@ +--- +title: Bucket Logging (Available Upon Request) +docId: 0191fc71-e031-761c-a16b-aa8ca9e44413 +metadata: + description: Detailed guide on enabling bucket logging + title: 'Bucket Logging (Available Upon Request)' +--- + +The `GetBucketLogging` and `PutBucketLogging` actions are not available, however, you can get the same functionality by following the steps below. + +{% callout type="info" %} +**Request Bucket Logging:** This feature is currently provided upon request - please submit your request here, and include "Enable Bucket Logging" as the subject: + +[Submit a support request](https://supportdcs.storj.io/hc/en-us/requests/new?ticket_form_id=360000379291) + +_Note: It may take up to two weeks to process your request._ +{% /callout %} + +## Enabling Bucket Logging + +To enable bucket logging, you will be asked to provide us with the following information via a secure channel: + +### Information Needed to Enable Logging + +| **Item** | **Details** | +|--------------------|----------------------------------------------------| +| **Satellite** | The Satellite your project is on: AP1, EU1, or US1 | +| **Project Name** | Your project's name | +| **Bucket Name(s)** | The bucket(s) to log | + +### Information About the Destination for Logs + +| **Item** | **Details** | +|------------------------------|------------------------------------------------------------| +| **Destination Project Name** | The project where logs will be stored | +| **Destination Bucket Name** | The bucket to store logs | +| **Prefix (optional)** | Prefix for log object keys | +| **Write-only Access Grant** | Access grant with write-only permissions (see steps below) | + +### Steps to Create a Write-Only Access Grant for Logging Destination + +{% callout type="info" %} +**Important:** Access grants used to access the watched bucket need to be created after June 25th 2024. +{% /callout %} + +1. **Generate a New Access Grant:** + + - Log in to the Satellite UI. + - Click **New Access Key** and select **"Access Grant"**. + - Name the access grant appropriately. + +2. **Select Advanced Options:** + + - On the second screen, click on **"Advanced Options"**. + - This allows you to customize permissions for the access grant. + +3. **Set Encryption Passphrase:** + + - Enter an encryption passphrase of your choice. + + {% callout type="info" %} + **Important:** Keep this passphrase secure. Losing it will prevent you from decrypting the log data. + {% /callout %} + +4. **Configure Permissions:** + + - On the permissions screen, select **"Write Only"**. + - Ensure no other permissions are granted. + - This restricts the access grant to only write logging files without the ability to read, delete, or overwrite them. + +5. **Limit Access to Destination Bucket:** + + - Specify the destination bucket for the logs. + - This limits the access grant to the specified bucket only. + +6. **Set Expiration (Optional):** + + - You can add an expiration date to the access. + + {% callout type="info" %} + **Recommendation:** Select **"No Expiration"** to ensure continuous logging. If the access expires, logging will stop. + {% /callout %} + +7. **Review and Create Access Grant:** + + - Confirm all selections are correct. + - Click on **"Create Access"** to generate the access grant. + +8. **Provide Access Grant to Storj:** + + - Send us the generated access grant over a secure channel. + +### Log Format + +The log objects are stored in the following key format with non-date-based partitioning: +``` +[DestinationPrefix][YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString] +``` + +**Example:** +``` +v-0730-ttl30/2024-08-29-03-48-32-33A6009CA7B144AF +``` + + +### Log Fields + +The fields in the logs conform to the [Amazon S3 Server Access Log Format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html#log-record-fields). + +### Example Logs + +``` +1831182b-718f-471f-852d-6e1a4701eadd v-0730-ttl30 [29/Aug/2024:03:07:14 +0000] 136.0.77.2 1831182b-718f-471f-852d-6e1a4701eadd 17F0142B99B6139E PostPolicyBucket - "POST /v-0730-ttl30/ HTTP/1.1" 204 - - - - - "-" "Go-http-client/1.1" - 46ccb4215d73986341ced57f4a224a18133bf183644e3873e3384d8f95295bb3 SigV4 TLS_AES_128_GCM_SHA256 - - TLS 1.3 - - +1831182b-718f-471f-852d-6e1a4701eadd v-0730-ttl30 [29/Aug/2024:03:07:14 +0000] 136.0.77.2 1831182b-718f-471f-852d-6e1a4701eadd 17F0142B9E85FFFE GetBucketLocation - "GET /v-0730-ttl30/?location= HTTP/1.1" 200 - 134 - - - "-" "MinIO (linux; amd64) minio-go/v7.0.70" - 46ccb4215d73986341ced57f4a224a18133bf183644e3873e3384d8f95295bb3 SigV4 TLS_AES_128_GCM_SHA256 - - TLS 1.3 - - +1831182b-718f-471f-852d-6e1a4701eadd v-0730-ttl30 [29/Aug/2024:03:07:14 +0000] 136.0.77.2 1831182b-718f-471f-852d-6e1a4701eadd 17F0142B9E845AFB GetBucketLocation - "GET /v-0730-ttl30/?location= HTTP/1.1" 200 - 134 - - - "-" "MinIO (linux; amd64) minio-go/v7.0.70" - 46ccb4215d73986341ced57f4a224a18133bf183644e3873e3384d8f95295bb3 SigV4 TLS_AES_128_GCM_SHA256 - - TLS 1.3 - - +1831182b-718f-471f-852d-6e1a4701eadd v-0730-ttl30 [29/Aug/2024:03:07:14 +0000] 136.0.77.2 1831182b-718f-471f-852d-6e1a4701eadd 17F0142B9992374E PostPolicyBucket - "POST /v-0730-ttl30/ HTTP/1.1" 204 - - - - - "-" "Go-http-client/1.1" - 46ccb4215d73986341ced57f4a224a18133bf183644e3873e3384d8f95295bb3 SigV4 TLS_AES_128_GCM_SHA256 - - TLS 1.3 - - +``` \ No newline at end of file diff --git a/app/dcs/buckets/cors/page.md b/app/(docs)/dcs/buckets/cors/page.md similarity index 100% rename from app/dcs/buckets/cors/page.md rename to app/(docs)/dcs/buckets/cors/page.md diff --git a/app/dcs/buckets/create-buckets/page.md b/app/(docs)/dcs/buckets/create-buckets/page.md similarity index 100% rename from app/dcs/buckets/create-buckets/page.md rename to app/(docs)/dcs/buckets/create-buckets/page.md diff --git a/app/dcs/buckets/data-location/page.md b/app/(docs)/dcs/buckets/data-location/page.md similarity index 100% rename from app/dcs/buckets/data-location/page.md rename to app/(docs)/dcs/buckets/data-location/page.md diff --git a/app/dcs/buckets/object-lifecycles/page.md b/app/(docs)/dcs/buckets/object-lifecycles/page.md similarity index 100% rename from app/dcs/buckets/object-lifecycles/page.md rename to app/(docs)/dcs/buckets/object-lifecycles/page.md diff --git a/app/dcs/buckets/object-listings/page.md b/app/(docs)/dcs/buckets/object-listings/page.md similarity index 100% rename from app/dcs/buckets/object-listings/page.md rename to app/(docs)/dcs/buckets/object-listings/page.md diff --git a/app/dcs/buckets/preferred-storage-region/page.md b/app/(docs)/dcs/buckets/preferred-storage-region/page.md similarity index 100% rename from app/dcs/buckets/preferred-storage-region/page.md rename to app/(docs)/dcs/buckets/preferred-storage-region/page.md diff --git a/app/dcs/buckets/public-buckets/page.md b/app/(docs)/dcs/buckets/public-buckets/page.md similarity index 100% rename from app/dcs/buckets/public-buckets/page.md rename to app/(docs)/dcs/buckets/public-buckets/page.md diff --git a/app/dcs/code/_meta.json b/app/(docs)/dcs/code/_meta.json similarity index 100% rename from app/dcs/code/_meta.json rename to app/(docs)/dcs/code/_meta.json diff --git a/app/dcs/code/aws/aws-cli-endpoint/page.md b/app/(docs)/dcs/code/aws/aws-cli-endpoint/page.md similarity index 100% rename from app/dcs/code/aws/aws-cli-endpoint/page.md rename to app/(docs)/dcs/code/aws/aws-cli-endpoint/page.md diff --git a/app/dcs/code/aws/nodejs/page.md b/app/(docs)/dcs/code/aws/nodejs/page.md similarity index 100% rename from app/dcs/code/aws/nodejs/page.md rename to app/(docs)/dcs/code/aws/nodejs/page.md diff --git a/app/dcs/code/partner-program-tools/page.md b/app/(docs)/dcs/code/partner-program-tools/page.md similarity index 100% rename from app/dcs/code/partner-program-tools/page.md rename to app/(docs)/dcs/code/partner-program-tools/page.md diff --git a/app/dcs/code/presigned-urls-serverless-cloud/page.md b/app/(docs)/dcs/code/presigned-urls-serverless-cloud/page.md similarity index 100% rename from app/dcs/code/presigned-urls-serverless-cloud/page.md rename to app/(docs)/dcs/code/presigned-urls-serverless-cloud/page.md diff --git a/app/dcs/code/rails-activestorage/page.md b/app/(docs)/dcs/code/rails-activestorage/page.md similarity index 100% rename from app/dcs/code/rails-activestorage/page.md rename to app/(docs)/dcs/code/rails-activestorage/page.md diff --git a/app/dcs/code/static-site-hosting/custom-domains/page.md b/app/(docs)/dcs/code/static-site-hosting/custom-domains/page.md similarity index 99% rename from app/dcs/code/static-site-hosting/custom-domains/page.md rename to app/(docs)/dcs/code/static-site-hosting/custom-domains/page.md index 58a96b5d2..219264bf5 100644 --- a/app/dcs/code/static-site-hosting/custom-domains/page.md +++ b/app/(docs)/dcs/code/static-site-hosting/custom-domains/page.md @@ -196,7 +196,7 @@ Here is an example the steps required to host a website on a custom domain (e.g. If you've already created a share at the root bucket, you must revoke that share to disallow access and recreate the share with the new restriction. {% /callout %} -1. Navigate to your custom domain (e.g ) +1. Navigate to your custom domain (e.g https://my-website.storj.dev/) ## Considerations if setting up DNS with a CDN like Cloudflare diff --git a/app/dcs/code/static-site-hosting/page.md b/app/(docs)/dcs/code/static-site-hosting/page.md similarity index 100% rename from app/dcs/code/static-site-hosting/page.md rename to app/(docs)/dcs/code/static-site-hosting/page.md diff --git a/app/dcs/getting-started/page.md b/app/(docs)/dcs/getting-started/page.md similarity index 100% rename from app/dcs/getting-started/page.md rename to app/(docs)/dcs/getting-started/page.md diff --git a/app/dcs/migrate/backblaze/page.md b/app/(docs)/dcs/migrate/backblaze/page.md similarity index 98% rename from app/dcs/migrate/backblaze/page.md rename to app/(docs)/dcs/migrate/backblaze/page.md index 33f40e833..8f91c3f81 100644 --- a/app/dcs/migrate/backblaze/page.md +++ b/app/(docs)/dcs/migrate/backblaze/page.md @@ -143,5 +143,3 @@ rclone check backblaze:my-backblaze-bucket storj:my-backblaze-bucket This command will compare the source (Backblaze) and destination (Storj) and report any discrepancies. You can also see the contents of your Backblaze bucket in the Storj Web Console. - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/b2_bucket_on_storj.png) diff --git a/app/dcs/migrate/page.md b/app/(docs)/dcs/migrate/page.md similarity index 100% rename from app/dcs/migrate/page.md rename to app/(docs)/dcs/migrate/page.md diff --git a/app/dcs/migrate/wasabi/page.md b/app/(docs)/dcs/migrate/wasabi/page.md similarity index 97% rename from app/dcs/migrate/wasabi/page.md rename to app/(docs)/dcs/migrate/wasabi/page.md index b55517340..a2812b0a9 100644 --- a/app/dcs/migrate/wasabi/page.md +++ b/app/(docs)/dcs/migrate/wasabi/page.md @@ -142,5 +142,3 @@ rclone check wasabi:my-wasabi-bucket storj:my-wasabi-bucket This command will compare the source (Wasabi) and destination (Storj) and report any discrepancies. You can also see the contents of your Wasabi bucket in the Storj Web Console. - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/k_hZRrlzb3x4CqXweWmoD_screenshot-2023-07-05-at-30729-pm.png) diff --git a/app/dcs/Objects/page.md b/app/(docs)/dcs/objects/page.md similarity index 77% rename from app/dcs/Objects/page.md rename to app/(docs)/dcs/objects/page.md index 0e177ea5f..334e2c78f 100644 --- a/app/dcs/Objects/page.md +++ b/app/(docs)/dcs/objects/page.md @@ -50,13 +50,11 @@ uplink cp ~/Downloads/storj-tree.png sj://my-bucket {% tab label="Storj Console" %} -1. Navigate to **Buckets** on the left side menu +1. Navigate to **Browse** on the left side menu -1. Select your bucket from the list +2. Open your bucket from the list -1. Select **Upload** - - ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/storj-console-upload.png) +3. Click **Upload** {% /tab %} @@ -98,15 +96,11 @@ uplink cp sj://my-bucket ~/Downloads/storj-tree.png {% tab label="Storj Console" %} -1. Navigate to **Buckets** on the left side menu - -1. Select your bucket from the list - -1. Click the 3-dots to show the additonal options menu +1. Navigate to **Browse** on the left side menu -1. Select **Download** +2. Open your bucket from the list - ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/storj-console-download.png) +3. Click on the Download button on your file. {% /tab %} diff --git a/app/dcs/pricing/page.md b/app/(docs)/dcs/pricing/page.md similarity index 93% rename from app/dcs/pricing/page.md rename to app/(docs)/dcs/pricing/page.md index 410646203..82b25aab9 100644 --- a/app/dcs/pricing/page.md +++ b/app/(docs)/dcs/pricing/page.md @@ -52,9 +52,9 @@ Unlike other cloud object storage vendors, we don't use high egress fees to crea For most users and most usage patterns, we do not expect a Per Segment Fee to be charged. Only when large numbers of segments are stored relative to a disproportionately small amount of data do we expect there to be a Per Segment Fee. Only use cases with large numbers of very small files or large numbers of very small Multipart Upload Parts are expected to be subject to the Per Segment Fee. -Each Segment stored on the network in excess of the default Segment Project Limit is charged a nominal Per Segment fee. The Per Segment Fee is $0.0000088 Per Segment Per Month. Distributed object storage is optimized for large files (several MB or larger - the larger the better). Very small objects generate more overhead with the storage of the metadata for the file than the actual storage of the object. Consequently, we charge a small Per Segment Fee to account for that overhead. If a user is storing even large numbers of big files, the Per Segment Fee will be inconsequential. If a user streams millions of small files, the Per Segment Fee will offset the cost associated with the metadata overhead. +Each Segment stored on the network is charged a nominal Per Segment fee. The Per Segment Fee is $0.0000088 Per Segment Per Month. Distributed object storage is optimized for large files (several MB or larger - the larger the better). Very small objects generate more overhead with the storage of the metadata for the file than the actual storage of the object. Consequently, we charge a small Per Segment Fee to account for that overhead. If a user is storing even large numbers of big files, the Per Segment Fee will be inconsequential. If a user streams millions of small files, the Per Segment Fee will offset the cost associated with the metadata overhead. -Thd segment Limit will be billed at a rate of $0.0000088 per Segment per month, equivalent to a rate of $0.00000001222 per Segment Hour. +The segment Limit will be billed at a rate of $0.0000088 per Segment per month, equivalent to a rate of $0.00000001222 per Segment Hour. The Storj Platform distinguishes between two types of object storage: remote and inline. Remote objects are large enough to erasure code and store the pieces of the file on storage nodes. Inline objects are smaller than the metadata associated with the actual object. In the case of objects that are smaller than the associated metadata, it is more efficient to store the object inline in the satellite metadata database. When storing a large number of tiny files, the best practice is to employ a packing strategy to store larger blocks of small files as a single object. @@ -109,7 +109,7 @@ Assuming a 1TB data set comprised of 1,000 1GB files is stored for an entire mon | Part Size | Files | Parts/ S | Segment Hours | Chargeable Segment Hours | Monthly Cost of Per Segment Fee | Monthly cost of storage | | :-------- | :---- | :------- | :------------ | :----------------------- | :------------------------------ | :---------------------- | | 64MB | 1,000 | 15,625 | 11.25M | 11.25M | $0.14 | $4.14 | -| 5MB | 1,000 | 200,000 | 144.0M | 108.0M | $1.32 | $5.32 | +| 5MB | 1,000 | 200,000 | 144.0M | 144.0M | $1.76 | $5.76 | **Multipart Cost Example 1:** diff --git a/app/dcs/third-party-tools/MASV/page.md b/app/(docs)/dcs/third-party-tools/MASV/page.md similarity index 100% rename from app/dcs/third-party-tools/MASV/page.md rename to app/(docs)/dcs/third-party-tools/MASV/page.md diff --git a/app/dcs/third-party-tools/acronis/page.md b/app/(docs)/dcs/third-party-tools/acronis/page.md similarity index 100% rename from app/dcs/third-party-tools/acronis/page.md rename to app/(docs)/dcs/third-party-tools/acronis/page.md diff --git a/app/dcs/third-party-tools/arq/page.md b/app/(docs)/dcs/third-party-tools/arq/page.md similarity index 100% rename from app/dcs/third-party-tools/arq/page.md rename to app/(docs)/dcs/third-party-tools/arq/page.md diff --git a/app/dcs/third-party-tools/atempo-miria/page.md b/app/(docs)/dcs/third-party-tools/atempo-miria/page.md similarity index 100% rename from app/dcs/third-party-tools/atempo-miria/page.md rename to app/(docs)/dcs/third-party-tools/atempo-miria/page.md diff --git a/app/dcs/third-party-tools/bacula/page.md b/app/(docs)/dcs/third-party-tools/bacula/page.md similarity index 100% rename from app/dcs/third-party-tools/bacula/page.md rename to app/(docs)/dcs/third-party-tools/bacula/page.md diff --git a/app/dcs/third-party-tools/bunny/page.md b/app/(docs)/dcs/third-party-tools/bunny/page.md similarity index 98% rename from app/dcs/third-party-tools/bunny/page.md rename to app/(docs)/dcs/third-party-tools/bunny/page.md index 63dc6f7fa..b3c3968ad 100644 --- a/app/dcs/third-party-tools/bunny/page.md +++ b/app/(docs)/dcs/third-party-tools/bunny/page.md @@ -81,3 +81,7 @@ Ensure you include the trailing `.` at the end of your CNAME if your DNS provide {% callout type="info" %} For more detailed information, refer to [Bunny's documentation for integrating Bunny CDN with Cloudflare](https://support.bunny.net/hc/en-us/articles/360001631951-How-to-set-up-BunnyCDN-with-a-custom-hostname-on-CloudFlare) {% /callout %} +<<<<<<< Updated upstream +======= + +>>>>>>> Stashed changes diff --git a/app/dcs/third-party-tools/chainstate-snapshots/page.md b/app/(docs)/dcs/third-party-tools/chainstate-snapshots/page.md similarity index 100% rename from app/dcs/third-party-tools/chainstate-snapshots/page.md rename to app/(docs)/dcs/third-party-tools/chainstate-snapshots/page.md diff --git a/app/dcs/third-party-tools/comet-backup/page.md b/app/(docs)/dcs/third-party-tools/comet-backup/page.md similarity index 100% rename from app/dcs/third-party-tools/comet-backup/page.md rename to app/(docs)/dcs/third-party-tools/comet-backup/page.md diff --git a/app/dcs/third-party-tools/commvault/page.md b/app/(docs)/dcs/third-party-tools/commvault/page.md similarity index 100% rename from app/dcs/third-party-tools/commvault/page.md rename to app/(docs)/dcs/third-party-tools/commvault/page.md diff --git a/app/dcs/third-party-tools/cyberduck/page.md b/app/(docs)/dcs/third-party-tools/cyberduck/page.md similarity index 100% rename from app/dcs/third-party-tools/cyberduck/page.md rename to app/(docs)/dcs/third-party-tools/cyberduck/page.md diff --git a/app/dcs/third-party-tools/dataverse/page.md b/app/(docs)/dcs/third-party-tools/dataverse/page.md similarity index 100% rename from app/dcs/third-party-tools/dataverse/page.md rename to app/(docs)/dcs/third-party-tools/dataverse/page.md diff --git a/app/dcs/third-party-tools/docker/page.md b/app/(docs)/dcs/third-party-tools/docker/page.md similarity index 100% rename from app/dcs/third-party-tools/docker/page.md rename to app/(docs)/dcs/third-party-tools/docker/page.md diff --git a/app/dcs/third-party-tools/duplicati/page.md b/app/(docs)/dcs/third-party-tools/duplicati/page.md similarity index 94% rename from app/dcs/third-party-tools/duplicati/page.md rename to app/(docs)/dcs/third-party-tools/duplicati/page.md index 1e42991f3..50199c7da 100644 --- a/app/dcs/third-party-tools/duplicati/page.md +++ b/app/(docs)/dcs/third-party-tools/duplicati/page.md @@ -21,11 +21,7 @@ To restore a small file, Duplicati has to download the entire block it is contai ## Install -{% callout type="info" %} -**Please note that the version used for writing this documentation is not yet released on the Duplicati homepage. Please download the canary version from** [**Duplicati Releases**](https://github.com/duplicati/duplicati/releases) **or use the canary** [docker container](https://hub.docker.com/r/duplicati/duplicati). -{% /callout %} - -1. [Download](https://github.com/duplicati/duplicati/releases) and install the Duplicati installer file for your OS or run the [docker container](https://hub.docker.com/r/duplicati/duplicati). Note **warning** above! +1. [Download](https://github.com/duplicati/duplicati/releases) and install the Duplicati installer file for your OS or run the [docker container](https://hub.docker.com/r/duplicati/duplicati). 2. Once installed, the software will open your browser to the local Duplicati dashboard. If not, it can be accessed at `http://localhost:8200/` diff --git a/app/dcs/third-party-tools/elements/page.md b/app/(docs)/dcs/third-party-tools/elements/page.md similarity index 100% rename from app/dcs/third-party-tools/elements/page.md rename to app/(docs)/dcs/third-party-tools/elements/page.md diff --git a/app/dcs/third-party-tools/fastly/page.md b/app/(docs)/dcs/third-party-tools/fastly/page.md similarity index 100% rename from app/dcs/third-party-tools/fastly/page.md rename to app/(docs)/dcs/third-party-tools/fastly/page.md diff --git a/app/dcs/third-party-tools/file-transfer-performance/page.md b/app/(docs)/dcs/third-party-tools/file-transfer-performance/page.md similarity index 100% rename from app/dcs/third-party-tools/file-transfer-performance/page.md rename to app/(docs)/dcs/third-party-tools/file-transfer-performance/page.md diff --git a/app/dcs/third-party-tools/filezilla/filezilla-native/page.md b/app/(docs)/dcs/third-party-tools/filezilla/filezilla-native/page.md similarity index 100% rename from app/dcs/third-party-tools/filezilla/filezilla-native/page.md rename to app/(docs)/dcs/third-party-tools/filezilla/filezilla-native/page.md diff --git a/app/dcs/third-party-tools/filezilla/filezilla-pro/page.md b/app/(docs)/dcs/third-party-tools/filezilla/filezilla-pro/page.md similarity index 100% rename from app/dcs/third-party-tools/filezilla/filezilla-pro/page.md rename to app/(docs)/dcs/third-party-tools/filezilla/filezilla-pro/page.md diff --git a/app/dcs/third-party-tools/filezilla/page.md b/app/(docs)/dcs/third-party-tools/filezilla/page.md similarity index 100% rename from app/dcs/third-party-tools/filezilla/page.md rename to app/(docs)/dcs/third-party-tools/filezilla/page.md diff --git a/app/dcs/third-party-tools/globus/page.md b/app/(docs)/dcs/third-party-tools/globus/page.md similarity index 100% rename from app/dcs/third-party-tools/globus/page.md rename to app/(docs)/dcs/third-party-tools/globus/page.md diff --git a/app/dcs/third-party-tools/hammerspace/page.md b/app/(docs)/dcs/third-party-tools/hammerspace/page.md similarity index 99% rename from app/dcs/third-party-tools/hammerspace/page.md rename to app/(docs)/dcs/third-party-tools/hammerspace/page.md index 2883f7cc3..52a33d752 100644 --- a/app/dcs/third-party-tools/hammerspace/page.md +++ b/app/(docs)/dcs/third-party-tools/hammerspace/page.md @@ -14,7 +14,7 @@ metadata: [**Hammerspace**](https://hammerspace.com/) is a software solution that creates a global data environment mimicking the experience of local access to globally distributed data. Data connected under Hammerspace can include different, typically incompatible storage types, such as data centers and cloud storage, as well as locations. Hammerspace provides a unified view and control over all connected data as a single, easily accessible dataset. -Contact Hammerspace for a [demo](https://hammerspace.com/hammerspace-software/). +For more information [contact Hammerspace](https://hammerspace.com/contact-us/) ## Advantages of Hammerspace with Storj diff --git a/app/dcs/third-party-tools/hashbackup/page.md b/app/(docs)/dcs/third-party-tools/hashbackup/page.md similarity index 100% rename from app/dcs/third-party-tools/hashbackup/page.md rename to app/(docs)/dcs/third-party-tools/hashbackup/page.md diff --git a/app/dcs/third-party-tools/hugging-face/page.md b/app/(docs)/dcs/third-party-tools/hugging-face/page.md similarity index 100% rename from app/dcs/third-party-tools/hugging-face/page.md rename to app/(docs)/dcs/third-party-tools/hugging-face/page.md diff --git a/app/dcs/third-party-tools/iconik/page.md b/app/(docs)/dcs/third-party-tools/iconik/page.md similarity index 100% rename from app/dcs/third-party-tools/iconik/page.md rename to app/(docs)/dcs/third-party-tools/iconik/page.md diff --git a/app/dcs/third-party-tools/ix-systems-truenas/page.md b/app/(docs)/dcs/third-party-tools/ix-systems-truenas/page.md similarity index 100% rename from app/dcs/third-party-tools/ix-systems-truenas/page.md rename to app/(docs)/dcs/third-party-tools/ix-systems-truenas/page.md diff --git a/app/dcs/third-party-tools/kerberos-vault/page.md b/app/(docs)/dcs/third-party-tools/kerberos-vault/page.md similarity index 100% rename from app/dcs/third-party-tools/kerberos-vault/page.md rename to app/(docs)/dcs/third-party-tools/kerberos-vault/page.md diff --git a/app/dcs/third-party-tools/livepeer/page.md b/app/(docs)/dcs/third-party-tools/livepeer/page.md similarity index 100% rename from app/dcs/third-party-tools/livepeer/page.md rename to app/(docs)/dcs/third-party-tools/livepeer/page.md diff --git a/app/dcs/third-party-tools/lucidlink/page.md b/app/(docs)/dcs/third-party-tools/lucidlink/page.md similarity index 100% rename from app/dcs/third-party-tools/lucidlink/page.md rename to app/(docs)/dcs/third-party-tools/lucidlink/page.md diff --git a/app/dcs/third-party-tools/mastodon/page.md b/app/(docs)/dcs/third-party-tools/mastodon/page.md similarity index 100% rename from app/dcs/third-party-tools/mastodon/page.md rename to app/(docs)/dcs/third-party-tools/mastodon/page.md diff --git a/app/dcs/third-party-tools/mongodb/page.md b/app/(docs)/dcs/third-party-tools/mongodb/page.md similarity index 100% rename from app/dcs/third-party-tools/mongodb/page.md rename to app/(docs)/dcs/third-party-tools/mongodb/page.md diff --git a/app/dcs/third-party-tools/mountainduck/page.md b/app/(docs)/dcs/third-party-tools/mountainduck/page.md similarity index 100% rename from app/dcs/third-party-tools/mountainduck/page.md rename to app/(docs)/dcs/third-party-tools/mountainduck/page.md diff --git a/app/dcs/third-party-tools/msp360/page.md b/app/(docs)/dcs/third-party-tools/msp360/page.md similarity index 100% rename from app/dcs/third-party-tools/msp360/page.md rename to app/(docs)/dcs/third-party-tools/msp360/page.md diff --git a/app/dcs/third-party-tools/nextcloud/page.md b/app/(docs)/dcs/third-party-tools/nextcloud/page.md similarity index 100% rename from app/dcs/third-party-tools/nextcloud/page.md rename to app/(docs)/dcs/third-party-tools/nextcloud/page.md diff --git a/app/dcs/third-party-tools/ocis/page.md b/app/(docs)/dcs/third-party-tools/ocis/page.md similarity index 98% rename from app/dcs/third-party-tools/ocis/page.md rename to app/(docs)/dcs/third-party-tools/ocis/page.md index e1c6da55b..c7ff61768 100644 --- a/app/dcs/third-party-tools/ocis/page.md +++ b/app/(docs)/dcs/third-party-tools/ocis/page.md @@ -155,4 +155,4 @@ STORAGE_USERS_S3NG_SECRET_KEY=secret_key # REPLACE ME STORAGE_USERS_S3NG_BUCKET=my-bucket # REPLACE ME ``` -For more information visit +For more information visit diff --git a/app/dcs/third-party-tools/opensea/page.md b/app/(docs)/dcs/third-party-tools/opensea/page.md similarity index 100% rename from app/dcs/third-party-tools/opensea/page.md rename to app/(docs)/dcs/third-party-tools/opensea/page.md diff --git a/app/dcs/third-party-tools/page.md b/app/(docs)/dcs/third-party-tools/page.md similarity index 62% rename from app/dcs/third-party-tools/page.md rename to app/(docs)/dcs/third-party-tools/page.md index da3c0384f..308c3f3ca 100644 --- a/app/dcs/third-party-tools/page.md +++ b/app/(docs)/dcs/third-party-tools/page.md @@ -18,30 +18,30 @@ Practical step-by-step guides to help you achieve a specific goal. Most useful w ## Backups -{% tag-links tag="backup" directory="./app/dcs/third-party-tools" %} +{% tag-links tag="backup" directory="./app/(docs)/dcs/third-party-tools" %} {% /tag-links %} ## Large Files -{% tag-links tag="large-file" directory="./app/dcs/third-party-tools" %} +{% tag-links tag="large-file" directory="./app/(docs)/dcs/third-party-tools" %} {% /tag-links %} ## File Management -{% tag-links tag="file-management" directory="./app/dcs/third-party-tools" %} +{% tag-links tag="file-management" directory="./app/(docs)/dcs/third-party-tools" %} {% /tag-links %} ## Content Delivery -{% tag-links tag="content-delivery" directory="./app/dcs/third-party-tools" %} +{% tag-links tag="content-delivery" directory="./app/(docs)/dcs/third-party-tools" %} {% /tag-links %} ## Scientific -{% tag-links tag="scientific" directory="./app/dcs/third-party-tools" %} +{% tag-links tag="scientific" directory="./app/(docs)/dcs/third-party-tools" %} {% /tag-links %} ## Cloud Ops -{% tag-links tag="cloud-ops" directory="./app/dcs/third-party-tools" %} +{% tag-links tag="cloud-ops" directory="./app/(docs)/dcs/third-party-tools" %} {% /tag-links %} diff --git a/app/dcs/third-party-tools/photos-plus/page.md b/app/(docs)/dcs/third-party-tools/photos-plus/page.md similarity index 100% rename from app/dcs/third-party-tools/photos-plus/page.md rename to app/(docs)/dcs/third-party-tools/photos-plus/page.md diff --git a/app/dcs/third-party-tools/pixelfed/page.md b/app/(docs)/dcs/third-party-tools/pixelfed/page.md similarity index 100% rename from app/dcs/third-party-tools/pixelfed/page.md rename to app/(docs)/dcs/third-party-tools/pixelfed/page.md diff --git a/app/dcs/third-party-tools/plex/page.md b/app/(docs)/dcs/third-party-tools/plex/page.md similarity index 100% rename from app/dcs/third-party-tools/plex/page.md rename to app/(docs)/dcs/third-party-tools/plex/page.md diff --git a/app/dcs/third-party-tools/qnap-hybrid-backup-sync-3/page.md b/app/(docs)/dcs/third-party-tools/qnap-hybrid-backup-sync-3/page.md similarity index 100% rename from app/dcs/third-party-tools/qnap-hybrid-backup-sync-3/page.md rename to app/(docs)/dcs/third-party-tools/qnap-hybrid-backup-sync-3/page.md diff --git a/app/dcs/third-party-tools/rclone/page.md b/app/(docs)/dcs/third-party-tools/rclone/page.md similarity index 100% rename from app/dcs/third-party-tools/rclone/page.md rename to app/(docs)/dcs/third-party-tools/rclone/page.md diff --git a/app/dcs/third-party-tools/rclone/rclone-native/page.md b/app/(docs)/dcs/third-party-tools/rclone/rclone-native/page.md similarity index 100% rename from app/dcs/third-party-tools/rclone/rclone-native/page.md rename to app/(docs)/dcs/third-party-tools/rclone/rclone-native/page.md diff --git a/app/dcs/third-party-tools/rclone/rclone-s3/page.md b/app/(docs)/dcs/third-party-tools/rclone/rclone-s3/page.md similarity index 100% rename from app/dcs/third-party-tools/rclone/rclone-s3/page.md rename to app/(docs)/dcs/third-party-tools/rclone/rclone-s3/page.md diff --git a/app/dcs/third-party-tools/restic/page.md b/app/(docs)/dcs/third-party-tools/restic/page.md similarity index 100% rename from app/dcs/third-party-tools/restic/page.md rename to app/(docs)/dcs/third-party-tools/restic/page.md diff --git a/app/dcs/third-party-tools/rubrik/page.md b/app/(docs)/dcs/third-party-tools/rubrik/page.md similarity index 100% rename from app/dcs/third-party-tools/rubrik/page.md rename to app/(docs)/dcs/third-party-tools/rubrik/page.md diff --git a/app/dcs/third-party-tools/rucio/page.md b/app/(docs)/dcs/third-party-tools/rucio/page.md similarity index 100% rename from app/dcs/third-party-tools/rucio/page.md rename to app/(docs)/dcs/third-party-tools/rucio/page.md diff --git a/app/dcs/third-party-tools/s3-browser/page.md b/app/(docs)/dcs/third-party-tools/s3-browser/page.md similarity index 100% rename from app/dcs/third-party-tools/s3-browser/page.md rename to app/(docs)/dcs/third-party-tools/s3-browser/page.md diff --git a/app/dcs/third-party-tools/s3fs/page.md b/app/(docs)/dcs/third-party-tools/s3fs/page.md similarity index 100% rename from app/dcs/third-party-tools/s3fs/page.md rename to app/(docs)/dcs/third-party-tools/s3fs/page.md diff --git a/app/dcs/third-party-tools/signiant/page.md b/app/(docs)/dcs/third-party-tools/signiant/page.md similarity index 100% rename from app/dcs/third-party-tools/signiant/page.md rename to app/(docs)/dcs/third-party-tools/signiant/page.md diff --git a/app/dcs/third-party-tools/splunk/page.md b/app/(docs)/dcs/third-party-tools/splunk/page.md similarity index 100% rename from app/dcs/third-party-tools/splunk/page.md rename to app/(docs)/dcs/third-party-tools/splunk/page.md diff --git a/app/dcs/third-party-tools/starfish/page.md b/app/(docs)/dcs/third-party-tools/starfish/page.md similarity index 100% rename from app/dcs/third-party-tools/starfish/page.md rename to app/(docs)/dcs/third-party-tools/starfish/page.md diff --git a/app/dcs/third-party-tools/tesla-sentry-mode-teslausb/page.md b/app/(docs)/dcs/third-party-tools/tesla-sentry-mode-teslausb/page.md similarity index 100% rename from app/dcs/third-party-tools/tesla-sentry-mode-teslausb/page.md rename to app/(docs)/dcs/third-party-tools/tesla-sentry-mode-teslausb/page.md diff --git a/app/dcs/third-party-tools/unitrends/page.md b/app/(docs)/dcs/third-party-tools/unitrends/page.md similarity index 100% rename from app/dcs/third-party-tools/unitrends/page.md rename to app/(docs)/dcs/third-party-tools/unitrends/page.md diff --git a/app/dcs/third-party-tools/veeam/page.md b/app/(docs)/dcs/third-party-tools/veeam/page.md similarity index 89% rename from app/dcs/third-party-tools/veeam/page.md rename to app/(docs)/dcs/third-party-tools/veeam/page.md index e2c54c0f5..c5ccdafd2 100644 --- a/app/dcs/third-party-tools/veeam/page.md +++ b/app/(docs)/dcs/third-party-tools/veeam/page.md @@ -33,7 +33,7 @@ To integrate Storj with Veeam, you will need to create S3 credentials in Storj a - A bucket for Veeam in your Storj instance. - An installation of Veeam. -{% callout %} Important Note: Please be sure to use at least **large** and ideally **extra large blocks** as demonstrated in the Job Creation Wizard below. {% /callout %} +{% callout %} Important Note: Please be sure to use at least **4MB** as demonstrated in the Job Creation Wizard below. {% /callout %} Download a [free trial](https://www.veeam.com/vm-backup-recovery-replication-software.html) of Veeam or [create a Veeam account](https://www.veeam.com/signin.html?client_id=my-veeam-com). @@ -97,7 +97,7 @@ Setting a value higher than 64 can increase throughput backing up (offloading) t 1. In the **Service point** field, specify an endpoint address of your S3 Compatible object storage. This will be the endpoint from the S3 credentials that you downloaded, and should be the following or similar: **https\://gateway.storjshare.io** -1. In the **Region** field, specify a region, such as **us-east-1**. +1. In the **Region** field, enter **storj**. 1. To add the Storj credentials, selecy the **Add...** button next to the **Credentials** drop-down list. Enter the access key and session key in their corresponding fields. Add an optional description in the **Description** field, if desired. {% callout type="info" %} @@ -115,11 +115,18 @@ Setting a value higher than 64 can increase throughput backing up (offloading) t #### Specify the bucket settings -1. From the **Bucket** drop-down list, select the "veeam" bucket created earlier. +1. From the **Bucket** drop-down list, select the bucket name created earlier. -1. In the **Select Folder** field, select **Browse** and find the cloud folder in your "veeam" bucket to map your object storage repository, if it already exists. If not, you can select **New Folder** to make a new one. +1. In the **Select Folder** field, select **Browse** and find the cloud folder in your bucket to map your object storage repository, if it already exists. If not, you can select **New Folder** to make a new one. 1. If desired, select the **Limit object storage consumption to** check box to define a soft limit that can be exceeded temporarily for your object storage consumption. Enter a limit value in terabytes or petabytes. + +1. If desired, select the **Make recent backups immutable for** check box to define a retention period. Enter a retention period in days. + {% callout type="info" %} + **Immutability Requirements** + + Veeam's immutability setting uses S3-Compatible Object Lock, which is currently in Beta. For more information and instructions on setting up your bucket with Object Lock, please refer to our documentation here: [](docId:gjrGzPNnhpYrAGTTAUaj). + {% /callout %} ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/YOE-le-vX4D0wfn7tnrey_archiverepositorys3cbucket.png) 1. Select **Next**. @@ -142,13 +149,7 @@ Use the New **Backup Job wizard** to configure the backup job. Follow the steps 3. Upon opening Advanced - Storage, you will be presented with the option of selecting Storage Optimization. Veeam recommends the default of 1MB because increasing the block size can result in larger incremental backups. -However, Storj's recommended setting for object storage is **4MB** or **8MB**. +However, Storj's recommended setting for object storage is **4MB**. Taking into account [Storj segment cost](docId:59T_2l7c1rvZVhI8p91VX#per-segment-fee), using larger block sizes both reduces overall Storj costs and provides better backup and restore times. -{% callout type="info" %} -To enable **8MB** as a Storage Optimization option, create a **UIShowLegacyBlockSize** (DWORD, 1) registry value under the `HKLM\SOFTWARE\Veeam\Veeam Backup and Replication` key on the backup server. This requires Veeam 11a or newer. - -You may need to reboot or restart the Veeam services for the change to take effect in Veeam's user interface. -{% /callout %} ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/veeam_advanced_settings.png) - diff --git a/app/dcs/third-party-tools/velero/page.md b/app/(docs)/dcs/third-party-tools/velero/page.md similarity index 100% rename from app/dcs/third-party-tools/velero/page.md rename to app/(docs)/dcs/third-party-tools/velero/page.md diff --git a/app/dcs/third-party-tools/wordpress-site-with-updraftplus/page.md b/app/(docs)/dcs/third-party-tools/wordpress-site-with-updraftplus/page.md similarity index 100% rename from app/dcs/third-party-tools/wordpress-site-with-updraftplus/page.md rename to app/(docs)/dcs/third-party-tools/wordpress-site-with-updraftplus/page.md diff --git a/app/dcs/third-party-tools/zerto/page.md b/app/(docs)/dcs/third-party-tools/zerto/page.md similarity index 100% rename from app/dcs/third-party-tools/zerto/page.md rename to app/(docs)/dcs/third-party-tools/zerto/page.md diff --git a/app/(docs)/layout.js b/app/(docs)/layout.js new file mode 100644 index 000000000..590a89416 --- /dev/null +++ b/app/(docs)/layout.js @@ -0,0 +1,47 @@ +import { Inter } from 'next/font/google' +import localFont from 'next/font/local' +import clsx from 'clsx' + +import { ThemeProvider } from '@/components/theme-provider' +import { Navigation } from '@/components/Navigation' +import { Hero as HeroWrap } from '@/components/Hero.client' +import { Hero } from '@/components/Hero' + +import '@/styles/tailwind.css' + +export const metadata = { + metadataBase: new URL(process.env.SITE_URL), + title: { + template: '%s - Storj Docs', + default: 'Storj Docs', + }, + description: 'Make the world your data center', + alternates: { + canonical: '/', + }, +} + +export default function RootLayout({ + // Layouts must accept a children prop. + // This will be populated with nested layouts or pages + children, +}) { + return ( + <> + + + +
+
+
+
+
+
+ +
+
+ {children} +
+ + ) +} diff --git a/app/learn/concepts/access/access-grants/api-key/page.md b/app/(docs)/learn/concepts/access/access-grants/api-key/page.md similarity index 100% rename from app/learn/concepts/access/access-grants/api-key/page.md rename to app/(docs)/learn/concepts/access/access-grants/api-key/page.md diff --git a/app/learn/concepts/access/access-grants/api-key/restriction/page.md b/app/(docs)/learn/concepts/access/access-grants/api-key/restriction/page.md similarity index 100% rename from app/learn/concepts/access/access-grants/api-key/restriction/page.md rename to app/(docs)/learn/concepts/access/access-grants/api-key/restriction/page.md diff --git a/app/learn/concepts/access/access-grants/page.md b/app/(docs)/learn/concepts/access/access-grants/page.md similarity index 86% rename from app/learn/concepts/access/access-grants/page.md rename to app/(docs)/learn/concepts/access/access-grants/page.md index 542cb83c5..10324184b 100644 --- a/app/learn/concepts/access/access-grants/page.md +++ b/app/(docs)/learn/concepts/access/access-grants/page.md @@ -11,7 +11,7 @@ metadata: and encryption or decryption. --- -An Access Grant is a bearer token that enables applications to interact with Storj to access objects stored on the service and decrypt them client-side. +An [Access Grant](docId:M-5oxBinC6J1D-qSNjKYS#access-grant) is a bearer token that enables applications to interact with Storj to access objects stored on the service and decrypt them client-side. An Access Grant is a security envelope that contains a satellite address, a restricted API Key, and a set of one or more restricted prefix-based encryption keys—everything an application needs to locate an object on the network, access that object, and decrypt it. @@ -23,7 +23,7 @@ Access Grants coordinate two parallel constructs—encryption and authorization Access Grants are used for access management for client applications using the libuplink library, the CLI, as well as for generating credentials for the S3 compatible gateway (both the hosted GatewayMT and the self-hosted GatewayST). {% /callout %} -To make the implementation of these constructs as easy as possible for developers, the Storj developer tools abstract the complexity of encoding objects for access management and encryption/decryption. A simple share command encapsulates an encryption key, an [](docId:XOtletuYWGeA2Om86yvwA) ( a bearer token), and the appropriate Satellite address into an encoded string called an Access Grant. +To make the implementation of these constructs as easy as possible for developers, the Storj developer tools abstract the complexity of encoding objects for access management and encryption/decryption. A simple share command encapsulates an [encryption key](docId:yI4q9JDB3w01xEkFWA4_z), an [](docId:XOtletuYWGeA2Om86yvwA) (a bearer token), and the appropriate Satellite address into an encoded string called an Access Grant. Access Grants can be imported easily into an Uplink client, whether it's the CLI, developer library, or a client application. Imported Access Grants are managed client-side and may be leveraged in applications via the uplink client library. diff --git a/app/learn/concepts/access/access-grants/when-to-use-the-satellite-web-interface-and-when-to-use-the-cli/page.md b/app/(docs)/learn/concepts/access/access-grants/when-to-use-the-satellite-web-interface-and-when-to-use-the-cli/page.md similarity index 100% rename from app/learn/concepts/access/access-grants/when-to-use-the-satellite-web-interface-and-when-to-use-the-cli/page.md rename to app/(docs)/learn/concepts/access/access-grants/when-to-use-the-satellite-web-interface-and-when-to-use-the-cli/page.md diff --git a/app/learn/concepts/access/access-management-at-the-edge/page.md b/app/(docs)/learn/concepts/access/access-management-at-the-edge/page.md similarity index 100% rename from app/learn/concepts/access/access-management-at-the-edge/page.md rename to app/(docs)/learn/concepts/access/access-management-at-the-edge/page.md diff --git a/app/learn/concepts/access/access-revocation/page.md b/app/(docs)/learn/concepts/access/access-revocation/page.md similarity index 100% rename from app/learn/concepts/access/access-revocation/page.md rename to app/(docs)/learn/concepts/access/access-revocation/page.md diff --git a/app/learn/concepts/access/capability-based-access-control/page.md b/app/(docs)/learn/concepts/access/capability-based-access-control/page.md similarity index 100% rename from app/learn/concepts/access/capability-based-access-control/page.md rename to app/(docs)/learn/concepts/access/capability-based-access-control/page.md diff --git a/app/learn/concepts/access/encryption-and-keys/key-management/page.md b/app/(docs)/learn/concepts/access/encryption-and-keys/key-management/page.md similarity index 100% rename from app/learn/concepts/access/encryption-and-keys/key-management/page.md rename to app/(docs)/learn/concepts/access/encryption-and-keys/key-management/page.md diff --git a/app/learn/concepts/access/encryption-and-keys/page.md b/app/(docs)/learn/concepts/access/encryption-and-keys/page.md similarity index 100% rename from app/learn/concepts/access/encryption-and-keys/page.md rename to app/(docs)/learn/concepts/access/encryption-and-keys/page.md diff --git a/app/learn/concepts/access/encryption-and-keys/when-to-use-different-encryption-keys/page.md b/app/(docs)/learn/concepts/access/encryption-and-keys/when-to-use-different-encryption-keys/page.md similarity index 100% rename from app/learn/concepts/access/encryption-and-keys/when-to-use-different-encryption-keys/page.md rename to app/(docs)/learn/concepts/access/encryption-and-keys/when-to-use-different-encryption-keys/page.md diff --git a/app/learn/concepts/access/page.md b/app/(docs)/learn/concepts/access/page.md similarity index 100% rename from app/learn/concepts/access/page.md rename to app/(docs)/learn/concepts/access/page.md diff --git a/app/learn/concepts/connectors/page.md b/app/(docs)/learn/concepts/connectors/page.md similarity index 100% rename from app/learn/concepts/connectors/page.md rename to app/(docs)/learn/concepts/connectors/page.md diff --git a/app/learn/concepts/consistency/page.md b/app/(docs)/learn/concepts/consistency/page.md similarity index 100% rename from app/learn/concepts/consistency/page.md rename to app/(docs)/learn/concepts/consistency/page.md diff --git a/app/learn/concepts/data-structure/page.md b/app/(docs)/learn/concepts/data-structure/page.md similarity index 100% rename from app/learn/concepts/data-structure/page.md rename to app/(docs)/learn/concepts/data-structure/page.md diff --git a/app/learn/concepts/decentralization/coordination-avoidance/page.md b/app/(docs)/learn/concepts/decentralization/coordination-avoidance/page.md similarity index 100% rename from app/learn/concepts/decentralization/coordination-avoidance/page.md rename to app/(docs)/learn/concepts/decentralization/coordination-avoidance/page.md diff --git a/app/learn/concepts/decentralization/page.md b/app/(docs)/learn/concepts/decentralization/page.md similarity index 100% rename from app/learn/concepts/decentralization/page.md rename to app/(docs)/learn/concepts/decentralization/page.md diff --git a/app/learn/concepts/definitions/page.md b/app/(docs)/learn/concepts/definitions/page.md similarity index 100% rename from app/learn/concepts/definitions/page.md rename to app/(docs)/learn/concepts/definitions/page.md diff --git a/app/learn/concepts/edge-services/auth-service/page.md b/app/(docs)/learn/concepts/edge-services/auth-service/page.md similarity index 100% rename from app/learn/concepts/edge-services/auth-service/page.md rename to app/(docs)/learn/concepts/edge-services/auth-service/page.md diff --git a/app/learn/concepts/edge-services/page.md b/app/(docs)/learn/concepts/edge-services/page.md similarity index 100% rename from app/learn/concepts/edge-services/page.md rename to app/(docs)/learn/concepts/edge-services/page.md diff --git a/app/learn/concepts/encryption-key/design-decision-end-to-end-encryption/page.md b/app/(docs)/learn/concepts/encryption-key/design-decision-end-to-end-encryption/page.md similarity index 100% rename from app/learn/concepts/encryption-key/design-decision-end-to-end-encryption/page.md rename to app/(docs)/learn/concepts/encryption-key/design-decision-end-to-end-encryption/page.md diff --git a/app/learn/concepts/encryption-key/design-decision-server-side-encryption/page.md b/app/(docs)/learn/concepts/encryption-key/design-decision-server-side-encryption/page.md similarity index 100% rename from app/learn/concepts/encryption-key/design-decision-server-side-encryption/page.md rename to app/(docs)/learn/concepts/encryption-key/design-decision-server-side-encryption/page.md diff --git a/app/learn/concepts/encryption-key/how-encryption-is-implemented/page.md b/app/(docs)/learn/concepts/encryption-key/how-encryption-is-implemented/page.md similarity index 100% rename from app/learn/concepts/encryption-key/how-encryption-is-implemented/page.md rename to app/(docs)/learn/concepts/encryption-key/how-encryption-is-implemented/page.md diff --git a/app/learn/concepts/encryption-key/page.md b/app/(docs)/learn/concepts/encryption-key/page.md similarity index 100% rename from app/learn/concepts/encryption-key/page.md rename to app/(docs)/learn/concepts/encryption-key/page.md diff --git a/app/learn/concepts/file-redundancy/page.md b/app/(docs)/learn/concepts/file-redundancy/page.md similarity index 100% rename from app/learn/concepts/file-redundancy/page.md rename to app/(docs)/learn/concepts/file-redundancy/page.md diff --git a/app/learn/concepts/file-repair/page.md b/app/(docs)/learn/concepts/file-repair/page.md similarity index 100% rename from app/learn/concepts/file-repair/page.md rename to app/(docs)/learn/concepts/file-repair/page.md diff --git a/app/learn/concepts/immutability/page.md b/app/(docs)/learn/concepts/immutability/page.md similarity index 100% rename from app/learn/concepts/immutability/page.md rename to app/(docs)/learn/concepts/immutability/page.md diff --git a/app/learn/concepts/key-architecture-constructs/page.md b/app/(docs)/learn/concepts/key-architecture-constructs/page.md similarity index 100% rename from app/learn/concepts/key-architecture-constructs/page.md rename to app/(docs)/learn/concepts/key-architecture-constructs/page.md diff --git a/app/learn/concepts/limits/page.md b/app/(docs)/learn/concepts/limits/page.md similarity index 95% rename from app/learn/concepts/limits/page.md rename to app/(docs)/learn/concepts/limits/page.md index c8fbbbc7e..9604a256a 100644 --- a/app/learn/concepts/limits/page.md +++ b/app/(docs)/learn/concepts/limits/page.md @@ -41,7 +41,7 @@ Adding a credit card as a payment method will result in your per-project limits Adding $10 or more worth of STORJ tokens to your account deposit address will automatically upgrade your account to PRO and you will also receive a bonus of 10% of the deposit amount on your balance. {% callout type="info" %} -Please note: The deposit address currently only accepts ERC20 STORJ token transactions on the Ethereum mainnet. +Please note: The deposit address currently only accepts transactions with ERC20 STORJ tokens on Ethereum mainnet or zkSync Era (note that **zkSync Lite is not supported**). {% /callout %} ## Rationales behind limits diff --git a/app/learn/concepts/linksharing-service/page.md b/app/(docs)/learn/concepts/linksharing-service/page.md similarity index 100% rename from app/learn/concepts/linksharing-service/page.md rename to app/(docs)/learn/concepts/linksharing-service/page.md diff --git a/app/learn/concepts/multi-tenant-data/multi-tenant-access-management/page.md b/app/(docs)/learn/concepts/multi-tenant-data/multi-tenant-access-management/page.md similarity index 100% rename from app/learn/concepts/multi-tenant-data/multi-tenant-access-management/page.md rename to app/(docs)/learn/concepts/multi-tenant-data/multi-tenant-access-management/page.md diff --git a/app/learn/concepts/multi-tenant-data/page.md b/app/(docs)/learn/concepts/multi-tenant-data/page.md similarity index 100% rename from app/learn/concepts/multi-tenant-data/page.md rename to app/(docs)/learn/concepts/multi-tenant-data/page.md diff --git a/app/learn/concepts/multiregion-availability/page.md b/app/(docs)/learn/concepts/multiregion-availability/page.md similarity index 100% rename from app/learn/concepts/multiregion-availability/page.md rename to app/(docs)/learn/concepts/multiregion-availability/page.md diff --git a/app/learn/concepts/s3-compatibility/page.md b/app/(docs)/learn/concepts/s3-compatibility/page.md similarity index 100% rename from app/learn/concepts/s3-compatibility/page.md rename to app/(docs)/learn/concepts/s3-compatibility/page.md diff --git a/app/learn/concepts/satellite/page.md b/app/(docs)/learn/concepts/satellite/page.md similarity index 100% rename from app/learn/concepts/satellite/page.md rename to app/(docs)/learn/concepts/satellite/page.md diff --git a/app/learn/concepts/security-models/page.md b/app/(docs)/learn/concepts/security-models/page.md similarity index 100% rename from app/learn/concepts/security-models/page.md rename to app/(docs)/learn/concepts/security-models/page.md diff --git a/app/learn/concepts/solution-architectures/common-architectural-patterns/page.md b/app/(docs)/learn/concepts/solution-architectures/common-architectural-patterns/page.md similarity index 100% rename from app/learn/concepts/solution-architectures/common-architectural-patterns/page.md rename to app/(docs)/learn/concepts/solution-architectures/common-architectural-patterns/page.md diff --git a/app/learn/concepts/solution-architectures/common-use-cases/page.md b/app/(docs)/learn/concepts/solution-architectures/common-use-cases/page.md similarity index 100% rename from app/learn/concepts/solution-architectures/common-use-cases/page.md rename to app/(docs)/learn/concepts/solution-architectures/common-use-cases/page.md diff --git a/app/learn/concepts/worm/page.md b/app/(docs)/learn/concepts/worm/page.md similarity index 100% rename from app/learn/concepts/worm/page.md rename to app/(docs)/learn/concepts/worm/page.md diff --git a/app/learn/page.md b/app/(docs)/learn/page.md similarity index 100% rename from app/learn/page.md rename to app/(docs)/learn/page.md diff --git a/app/learn/self-host/gateway-st/page.md b/app/(docs)/learn/self-host/gateway-st/page.md similarity index 100% rename from app/learn/self-host/gateway-st/page.md rename to app/(docs)/learn/self-host/gateway-st/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/generate-access-grants-and-tokens/generate-a-token/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/generate-access-grants-and-tokens/generate-a-token/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/generate-access-grants-and-tokens/generate-a-token/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/generate-access-grants-and-tokens/generate-a-token/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/generate-access-grants-and-tokens/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/generate-access-grants-and-tokens/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/generate-access-grants-and-tokens/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/generate-access-grants-and-tokens/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/delete-an-object/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/delete-an-object/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/delete-an-object/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/delete-an-object/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/download-an-object/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/download-an-object/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/download-an-object/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/download-an-object/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/list-an-object/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/list-an-object/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/list-an-object/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/list-an-object/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/interacting-with-your-first-object/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/generate-access/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/generate-access/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/generate-access/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/generate-access/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/import-access/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/import-access/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/import-access/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/import-access/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/revoke-an-access-to-an-object/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/revoke-an-access-to-an-object/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/revoke-an-access-to-an-object/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/sharing-your-first-object/revoke-an-access-to-an-object/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/create-a-bucket/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/create-a-bucket/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/create-a-bucket/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/create-a-bucket/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/create-first-access-grant/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/create-first-access-grant/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/create-first-access-grant/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/create-first-access-grant/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/set-up-uplink-cli/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/set-up-uplink-cli/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/set-up-uplink-cli/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/set-up-uplink-cli/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/upload-an-object/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/upload-an-object/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/upload-an-object/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/upload-an-object/page.md diff --git a/app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/view-distribution-of-an-object/page.md b/app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/view-distribution-of-an-object/page.md similarity index 100% rename from app/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/view-distribution-of-an-object/page.md rename to app/(docs)/learn/tutorials/quickstart-uplink-cli/uploading-your-first-object/view-distribution-of-an-object/page.md diff --git a/app/node/commercial-node/dashboard/page.md b/app/(docs)/node/commercial-node/dashboard/page.md similarity index 100% rename from app/node/commercial-node/dashboard/page.md rename to app/(docs)/node/commercial-node/dashboard/page.md diff --git a/app/node/commercial-node/page.md b/app/(docs)/node/commercial-node/page.md similarity index 100% rename from app/node/commercial-node/page.md rename to app/(docs)/node/commercial-node/page.md diff --git a/app/node/commercial-node/setup/ansible/page.md b/app/(docs)/node/commercial-node/setup/ansible/page.md similarity index 100% rename from app/node/commercial-node/setup/ansible/page.md rename to app/(docs)/node/commercial-node/setup/ansible/page.md diff --git a/app/node/commercial-node/setup/page.md b/app/(docs)/node/commercial-node/setup/page.md similarity index 100% rename from app/node/commercial-node/setup/page.md rename to app/(docs)/node/commercial-node/setup/page.md diff --git a/app/node/concepts/page.md b/app/(docs)/node/concepts/page.md similarity index 100% rename from app/node/concepts/page.md rename to app/(docs)/node/concepts/page.md diff --git a/app/node/faq/check-logs/page.md b/app/(docs)/node/faq/check-logs/page.md similarity index 100% rename from app/node/faq/check-logs/page.md rename to app/(docs)/node/faq/check-logs/page.md diff --git a/app/node/faq/check-my-node/page.md b/app/(docs)/node/faq/check-my-node/page.md similarity index 100% rename from app/node/faq/check-my-node/page.md rename to app/(docs)/node/faq/check-my-node/page.md diff --git a/app/node/faq/cli-on-windows-mac/page.md b/app/(docs)/node/faq/cli-on-windows-mac/page.md similarity index 100% rename from app/node/faq/cli-on-windows-mac/page.md rename to app/(docs)/node/faq/cli-on-windows-mac/page.md diff --git a/app/node/faq/estimate-payouts/page.md b/app/(docs)/node/faq/estimate-payouts/page.md similarity index 100% rename from app/node/faq/estimate-payouts/page.md rename to app/(docs)/node/faq/estimate-payouts/page.md diff --git a/app/node/faq/held-back-amount/page.md b/app/(docs)/node/faq/held-back-amount/page.md similarity index 100% rename from app/node/faq/held-back-amount/page.md rename to app/(docs)/node/faq/held-back-amount/page.md diff --git a/app/node/faq/how-do-i-change-my-parameters-such-as-payout-address-allotted-storage-space-and-bandwidth/page.md b/app/(docs)/node/faq/how-do-i-change-my-parameters-such-as-payout-address-allotted-storage-space-and-bandwidth/page.md similarity index 100% rename from app/node/faq/how-do-i-change-my-parameters-such-as-payout-address-allotted-storage-space-and-bandwidth/page.md rename to app/(docs)/node/faq/how-do-i-change-my-parameters-such-as-payout-address-allotted-storage-space-and-bandwidth/page.md diff --git a/app/node/faq/how-do-i-check-my-l2-payouts/page.md b/app/(docs)/node/faq/how-do-i-check-my-l2-payouts/page.md similarity index 100% rename from app/node/faq/how-do-i-check-my-l2-payouts/page.md rename to app/(docs)/node/faq/how-do-i-check-my-l2-payouts/page.md diff --git a/app/node/faq/how-do-i-estimate-my-potential-earnings-for-given-amount-of-space-and-bandwidth/page.md b/app/(docs)/node/faq/how-do-i-estimate-my-potential-earnings-for-given-amount-of-space-and-bandwidth/page.md similarity index 100% rename from app/node/faq/how-do-i-estimate-my-potential-earnings-for-given-amount-of-space-and-bandwidth/page.md rename to app/(docs)/node/faq/how-do-i-estimate-my-potential-earnings-for-given-amount-of-space-and-bandwidth/page.md diff --git a/app/(docs)/node/faq/how-do-i-know-the-exchange-rate-for-my-payout/page.md b/app/(docs)/node/faq/how-do-i-know-the-exchange-rate-for-my-payout/page.md new file mode 100644 index 000000000..c0a10313e --- /dev/null +++ b/app/(docs)/node/faq/how-do-i-know-the-exchange-rate-for-my-payout/page.md @@ -0,0 +1,15 @@ +--- +title: >- + How do I know the exchange rate for my payout? +docId: 6xwcyBYTMDNojI58mxXSd +metadata: + description: How to check the exchange rate for your payout +redirects: + - >- + /node/resources/faq/how-do-i-know-the-exchange-rate-for-my-payout + - /node/resources/faq/costs-basis-tool +--- + +If you want to know the amount of USD a particular storage node payment transaction was considered denominated in, please check out our cost basis tool. You’ll need to enter your payment transaction: + +[https://costbasis.storj.tools/](https://costbasis.storj.tools/) diff --git a/app/node/faq/how-the-online-score-is-calculated/page.md b/app/(docs)/node/faq/how-the-online-score-is-calculated/page.md similarity index 100% rename from app/node/faq/how-the-online-score-is-calculated/page.md rename to app/(docs)/node/faq/how-the-online-score-is-calculated/page.md diff --git a/app/node/faq/how-to-add-an-additional-drive/page.md b/app/(docs)/node/faq/how-to-add-an-additional-drive/page.md similarity index 100% rename from app/node/faq/how-to-add-an-additional-drive/page.md rename to app/(docs)/node/faq/how-to-add-an-additional-drive/page.md diff --git a/app/node/faq/how-to-remote-access-the-web-dashboard/page.md b/app/(docs)/node/faq/how-to-remote-access-the-web-dashboard/page.md similarity index 100% rename from app/node/faq/how-to-remote-access-the-web-dashboard/page.md rename to app/(docs)/node/faq/how-to-remote-access-the-web-dashboard/page.md diff --git a/app/node/faq/linux-static-mount/page.md b/app/(docs)/node/faq/linux-static-mount/page.md similarity index 100% rename from app/node/faq/linux-static-mount/page.md rename to app/(docs)/node/faq/linux-static-mount/page.md diff --git a/app/node/faq/low-payouts/page.md b/app/(docs)/node/faq/low-payouts/page.md similarity index 100% rename from app/node/faq/low-payouts/page.md rename to app/(docs)/node/faq/low-payouts/page.md diff --git a/app/node/faq/machine-restart-shutdown/page.md b/app/(docs)/node/faq/machine-restart-shutdown/page.md similarity index 100% rename from app/node/faq/machine-restart-shutdown/page.md rename to app/(docs)/node/faq/machine-restart-shutdown/page.md diff --git a/app/node/faq/migrate-my-node/how-to-migrate-the-windows-gui-node-from-a-one-physical-location-to-other/page.md b/app/(docs)/node/faq/migrate-my-node/how-to-migrate-the-windows-gui-node-from-a-one-physical-location-to-other/page.md similarity index 100% rename from app/node/faq/migrate-my-node/how-to-migrate-the-windows-gui-node-from-a-one-physical-location-to-other/page.md rename to app/(docs)/node/faq/migrate-my-node/how-to-migrate-the-windows-gui-node-from-a-one-physical-location-to-other/page.md diff --git a/app/node/faq/migrate-my-node/migrating-from-docker-cli-to-a-gui-install-on-windows/page.md b/app/(docs)/node/faq/migrate-my-node/migrating-from-docker-cli-to-a-gui-install-on-windows/page.md similarity index 100% rename from app/node/faq/migrate-my-node/migrating-from-docker-cli-to-a-gui-install-on-windows/page.md rename to app/(docs)/node/faq/migrate-my-node/migrating-from-docker-cli-to-a-gui-install-on-windows/page.md diff --git a/app/node/faq/migrate-my-node/migrating-from-windows-gui-installation-to-a-docker-cli/page.md b/app/(docs)/node/faq/migrate-my-node/migrating-from-windows-gui-installation-to-a-docker-cli/page.md similarity index 100% rename from app/node/faq/migrate-my-node/migrating-from-windows-gui-installation-to-a-docker-cli/page.md rename to app/(docs)/node/faq/migrate-my-node/migrating-from-windows-gui-installation-to-a-docker-cli/page.md diff --git a/app/node/faq/migrate-my-node/page.md b/app/(docs)/node/faq/migrate-my-node/page.md similarity index 100% rename from app/node/faq/migrate-my-node/page.md rename to app/(docs)/node/faq/migrate-my-node/page.md diff --git a/app/node/faq/other-commands/page.md b/app/(docs)/node/faq/other-commands/page.md similarity index 100% rename from app/node/faq/other-commands/page.md rename to app/(docs)/node/faq/other-commands/page.md diff --git a/app/node/faq/page.md b/app/(docs)/node/faq/page.md similarity index 97% rename from app/node/faq/page.md rename to app/(docs)/node/faq/page.md index 16c79f807..1c470241e 100644 --- a/app/node/faq/page.md +++ b/app/(docs)/node/faq/page.md @@ -44,6 +44,8 @@ metadata: [](docId:ADB7HqQRe45givmFK7bfI) +[](docId:6xwcyBYTMDNojI58mxXSd) + ### Remote access [](docId:pueo_P_wgMERT0DdEn2pr) diff --git a/app/node/faq/redirect-logs/page.md b/app/(docs)/node/faq/redirect-logs/page.md similarity index 100% rename from app/node/faq/redirect-logs/page.md rename to app/(docs)/node/faq/redirect-logs/page.md diff --git a/app/node/faq/remote-connection/page.md b/app/(docs)/node/faq/remote-connection/page.md similarity index 100% rename from app/node/faq/remote-connection/page.md rename to app/(docs)/node/faq/remote-connection/page.md diff --git a/app/node/faq/storage-bandwidth-usage/page.md b/app/(docs)/node/faq/storage-bandwidth-usage/page.md similarity index 100% rename from app/node/faq/storage-bandwidth-usage/page.md rename to app/(docs)/node/faq/storage-bandwidth-usage/page.md diff --git a/app/node/faq/storing-more-data/page.md b/app/(docs)/node/faq/storing-more-data/page.md similarity index 96% rename from app/node/faq/storing-more-data/page.md rename to app/(docs)/node/faq/storing-more-data/page.md index 68a9f22b3..b27f556cb 100644 --- a/app/node/faq/storing-more-data/page.md +++ b/app/(docs)/node/faq/storing-more-data/page.md @@ -5,7 +5,7 @@ redirects: - /node/resources/faq/storing-more-data --- -The most important aspect to increase the amount of data stored on your Node (and thus maximizing payout) is to build reputation of your Node’s ID over an extended period of time**.** +The most important aspect to increase the amount of data stored on your Node (and thus maximizing payout) is to build reputation of your Node’s ID over an extended period of time. When a Node first joins the network, there is a probationary period, during which the Node has to prove itself (e.g. maintaining a certain uptime and performance levels, passing all content audits). During that vetting period, the Node only receives as small amount of non-critical data (but still gets paid for this data). Once vetted, a Node can start receiving more data (and not just test data), but must continue to maintain uptime and audit requirements to avoid disqualification. diff --git a/app/node/faq/system-maintenance/page.md b/app/(docs)/node/faq/system-maintenance/page.md similarity index 100% rename from app/node/faq/system-maintenance/page.md rename to app/(docs)/node/faq/system-maintenance/page.md diff --git a/app/node/faq/where-can-i-check-for-a-new-version/page.md b/app/(docs)/node/faq/where-can-i-check-for-a-new-version/page.md similarity index 100% rename from app/node/faq/where-can-i-check-for-a-new-version/page.md rename to app/(docs)/node/faq/where-can-i-check-for-a-new-version/page.md diff --git a/app/node/faq/where-can-i-find-a-config-yaml/page.md b/app/(docs)/node/faq/where-can-i-find-a-config-yaml/page.md similarity index 100% rename from app/node/faq/where-can-i-find-a-config-yaml/page.md rename to app/(docs)/node/faq/where-can-i-find-a-config-yaml/page.md diff --git a/app/node/get-started/_meta.json b/app/(docs)/node/get-started/_meta.json similarity index 100% rename from app/node/get-started/_meta.json rename to app/(docs)/node/get-started/_meta.json diff --git a/app/node/get-started/auth-token/page.md b/app/(docs)/node/get-started/auth-token/page.md similarity index 100% rename from app/node/get-started/auth-token/page.md rename to app/(docs)/node/get-started/auth-token/page.md diff --git a/app/node/get-started/identity/page.md b/app/(docs)/node/get-started/identity/page.md similarity index 100% rename from app/node/get-started/identity/page.md rename to app/(docs)/node/get-started/identity/page.md diff --git a/app/node/get-started/install-node-software/cli/dashboard-cli/page.md b/app/(docs)/node/get-started/install-node-software/cli/dashboard-cli/page.md similarity index 100% rename from app/node/get-started/install-node-software/cli/dashboard-cli/page.md rename to app/(docs)/node/get-started/install-node-software/cli/dashboard-cli/page.md diff --git a/app/node/get-started/install-node-software/cli/docker/page.md b/app/(docs)/node/get-started/install-node-software/cli/docker/page.md similarity index 100% rename from app/node/get-started/install-node-software/cli/docker/page.md rename to app/(docs)/node/get-started/install-node-software/cli/docker/page.md diff --git a/app/node/get-started/install-node-software/cli/page.md b/app/(docs)/node/get-started/install-node-software/cli/page.md similarity index 100% rename from app/node/get-started/install-node-software/cli/page.md rename to app/(docs)/node/get-started/install-node-software/cli/page.md diff --git a/app/node/get-started/install-node-software/cli/software-updates/page.md b/app/(docs)/node/get-started/install-node-software/cli/software-updates/page.md similarity index 100% rename from app/node/get-started/install-node-software/cli/software-updates/page.md rename to app/(docs)/node/get-started/install-node-software/cli/software-updates/page.md diff --git a/app/node/get-started/install-node-software/cli/storage-node/page.md b/app/(docs)/node/get-started/install-node-software/cli/storage-node/page.md similarity index 88% rename from app/node/get-started/install-node-software/cli/storage-node/page.md rename to app/(docs)/node/get-started/install-node-software/cli/storage-node/page.md index 5949cfbb8..d01afdb1e 100644 --- a/app/node/get-started/install-node-software/cli/storage-node/page.md +++ b/app/(docs)/node/get-started/install-node-software/cli/storage-node/page.md @@ -34,7 +34,17 @@ The writeability and readability checks are performed on the storage location, n {% tabs %} {% tab label="Linux" %} {% callout type="danger" %} -**Linux Users:** You **must** static mount via /etc/fstab. Failure to do so will put you in high risk of failing audits and getting disqualified. Here's how to do that: [](docId:nZeFxmawYPdgkwUPy6f9s) +**Linux Users:** You **must** static mount via /etc/fstab. Failure to do so will put you in high risk of failing audits and getting disqualified. The mount options must include `exec` permission. Here's how to do that: [](docId:nZeFxmawYPdgkwUPy6f9s) +{% /callout %} + +{% callout type="info" %} +You need to allow an execution for the `bin` subfolder in your storage location. The disk mount options must include `exec` permission. +Replace the `` with your parameter. + +```shell +mkdir -p /bin +chmod +x /bin +``` {% /callout %} 1. Copy the command into a plain text editor like a `nano`: @@ -51,6 +61,16 @@ docker run --rm -e SETUP="true" \ {% tab label="macOS" %} +{% callout type="info" %} +You need to allow an execution for the `bin` subfolder in your storage location. +Replace the `` with your parameter. + +```shell +mkdir -p /bin +chmod +x /bin +``` +{% /callout %} + 1. Copy the command into a plain text editor (do not use any word processors include Notes): ```shell diff --git a/app/node/get-started/install-node-software/gui-windows/dashboard/page.md b/app/(docs)/node/get-started/install-node-software/gui-windows/dashboard/page.md similarity index 100% rename from app/node/get-started/install-node-software/gui-windows/dashboard/page.md rename to app/(docs)/node/get-started/install-node-software/gui-windows/dashboard/page.md diff --git a/app/node/get-started/install-node-software/gui-windows/page.md b/app/(docs)/node/get-started/install-node-software/gui-windows/page.md similarity index 100% rename from app/node/get-started/install-node-software/gui-windows/page.md rename to app/(docs)/node/get-started/install-node-software/gui-windows/page.md diff --git a/app/node/get-started/install-node-software/gui-windows/storage-node/page.md b/app/(docs)/node/get-started/install-node-software/gui-windows/storage-node/page.md similarity index 100% rename from app/node/get-started/install-node-software/gui-windows/storage-node/page.md rename to app/(docs)/node/get-started/install-node-software/gui-windows/storage-node/page.md diff --git a/app/node/get-started/install-node-software/page.md b/app/(docs)/node/get-started/install-node-software/page.md similarity index 100% rename from app/node/get-started/install-node-software/page.md rename to app/(docs)/node/get-started/install-node-software/page.md diff --git a/app/node/get-started/install-node-software/qnap-storage-node-app/page.md b/app/(docs)/node/get-started/install-node-software/qnap-storage-node-app/page.md similarity index 100% rename from app/node/get-started/install-node-software/qnap-storage-node-app/page.md rename to app/(docs)/node/get-started/install-node-software/qnap-storage-node-app/page.md diff --git a/app/node/get-started/port-forwarding/page.md b/app/(docs)/node/get-started/port-forwarding/page.md similarity index 100% rename from app/node/get-started/port-forwarding/page.md rename to app/(docs)/node/get-started/port-forwarding/page.md diff --git a/app/node/get-started/prerequisites/page.md b/app/(docs)/node/get-started/prerequisites/page.md similarity index 91% rename from app/node/get-started/prerequisites/page.md rename to app/(docs)/node/get-started/prerequisites/page.md index 2e544d0a6..c807fc9ba 100644 --- a/app/node/get-started/prerequisites/page.md +++ b/app/(docs)/node/get-started/prerequisites/page.md @@ -12,9 +12,9 @@ Proceeding constitutes acceptance of our [Terms and conditions](https://www.stor ### Recommended * One processor core for each storage node process -* One hard drive, JBOD, per storage node process. NO SMR. +* One hard drive per storage node process. NO SMR. Connect drives without RAID controllers OR configure the RAID controller to passthrough/IT mode. * 2 TB of available space per storage node process -* 1.5 TB of transit per TB of storage node capacity; unlimited preferred +* 1.5 TB per month of transit per TB of storage node capacity; unlimited preferred * 3 Mbps upload bandwidth per TB of capacity * 5 Mbps download bandwidth per TB of capacity * Uptime (online and operational) of 99.5% per month @@ -22,9 +22,9 @@ Proceeding constitutes acceptance of our [Terms and conditions](https://www.stor ### Minimum * One processor core for each storage node process -* One hard drive, JBOD, per storage node process. NO SMR. +* One hard drive per storage node process. NO SMR. Connect drives without RAID controllers OR configure the RAID controller to passthrough/IT mode. * 500 GB of available space per storage node process -* 1.5 TB of transit per TB of storage node capacity +* 1.5 TB per month of transit per TB of storage node capacity * 1 Mbps upload bandwidth per TB of capacity * 3 Mbps download bandwidth per TB of capacity * Uptime (online and operational) of 99.3% per month, max total downtime of 5 hours monthly @@ -33,7 +33,7 @@ Proceeding constitutes acceptance of our [Terms and conditions](https://www.stor {% tabs %} {% tab label="Linux (Preferred)" %} -CentOS - A maintained version of CentOS 7 +CentOS - A maintained version of CentOS Debian - 64-bit version of one of these Debian or Raspbian versions: @@ -48,7 +48,7 @@ Ubuntu - 64-bit version of one of these Ubuntu versions: - Bionic 18.04 (LTS) or later > **Make sure you use static mount for your hard drive via** -> **/etc/fstab**: +> **/etc/fstab and drive is mounted with `exec` permission**: > See [](docId:nZeFxmawYPdgkwUPy6f9s). {% /tab %} @@ -64,7 +64,7 @@ VirtualBox is supported (version 4.3.30 and up) {% /tab %} {% tab label="Windows" %} -Windows 8, Windows Server 2012 or later. +Windows 10, Windows Server 2016 or later. **If you are currently running a storage node on Windows using the Docker desktop, it will require monitoring. If you are still running a node with Docker, your node may go offline randomly and require restarting your node, so it is recommended you switch to the** [](docId:5shJebpS3baWj6LDV5ANQ). [](docId:jA6Jl8XzCR1nc4_WyJj1a) diff --git a/app/node/get-started/quic-requirements/linux-configuration-for-udp/page.md b/app/(docs)/node/get-started/quic-requirements/linux-configuration-for-udp/page.md similarity index 100% rename from app/node/get-started/quic-requirements/linux-configuration-for-udp/page.md rename to app/(docs)/node/get-started/quic-requirements/linux-configuration-for-udp/page.md diff --git a/app/node/get-started/quic-requirements/macosfreebsd-configuration-for-udp/page.md b/app/(docs)/node/get-started/quic-requirements/macosfreebsd-configuration-for-udp/page.md similarity index 100% rename from app/node/get-started/quic-requirements/macosfreebsd-configuration-for-udp/page.md rename to app/(docs)/node/get-started/quic-requirements/macosfreebsd-configuration-for-udp/page.md diff --git a/app/node/get-started/quic-requirements/page.md b/app/(docs)/node/get-started/quic-requirements/page.md similarity index 100% rename from app/node/get-started/quic-requirements/page.md rename to app/(docs)/node/get-started/quic-requirements/page.md diff --git a/app/node/get-started/setup/page.md b/app/(docs)/node/get-started/setup/page.md similarity index 100% rename from app/node/get-started/setup/page.md rename to app/(docs)/node/get-started/setup/page.md diff --git a/app/node/page.md b/app/(docs)/node/page.md similarity index 100% rename from app/node/page.md rename to app/(docs)/node/page.md diff --git a/app/node/payouts/page.md b/app/(docs)/node/payouts/page.md similarity index 98% rename from app/node/payouts/page.md rename to app/(docs)/node/payouts/page.md index c78eb216d..abbb5552a 100644 --- a/app/node/payouts/page.md +++ b/app/(docs)/node/payouts/page.md @@ -39,7 +39,7 @@ Storage Node Fees will not be paid for the following Storage Node usage: The following table includes the current Storj Satellite payout rates. | **Payment Category** | **Rates as of Dec 1st, 2023** | -| --------------------- | ----------------------------- | +| -------------------------- | ----------------------------- | | Storage (per TB per Month) | $1.50 | | Egress (per TB) | $2.00 | | Audit/Repair (per TB) | $2.00 | @@ -54,6 +54,8 @@ For a detailed understanding of how TB is defined, please see [](docId:59T_2l7c1 [How do I estimate my potential earnings for a given amount of space and bandwidth?](docId:bG8Q88XbTvEPkzsuc02T8) +[](docId:6xwcyBYTMDNojI58mxXSd) + ## Minimum payment thresholds All Storage Node payouts are subject to a per-wallet minimum threshold. We will not send a transaction where the fee for the transaction is more than 25% of the value of the transaction. The minimum threshold is calculated based on the average transaction fee value in USD from the previous 12 hours at the beginning of the payout process. For example, if the average transaction fee is the equivalent of $12.50, we’ll pay out all wallet addresses that have earned $50.00 and above. diff --git a/app/(docs)/node/payouts/zk-sync-opt-in-for-snos/page.md b/app/(docs)/node/payouts/zk-sync-opt-in-for-snos/page.md new file mode 100644 index 000000000..b743e6f9c --- /dev/null +++ b/app/(docs)/node/payouts/zk-sync-opt-in-for-snos/page.md @@ -0,0 +1,119 @@ +--- +title: ZkSync Payments +docId: 6TX_ve1PyUrXuwax-mWWw +redirects: + - >- + /node/dependencies/storage-node-operator-payout-information/zk-sync-opt-in-for-snos +--- + +How to configure L2 Payments with zkSync Era + +## Background + +L2 scaling aligns with our goal of bringing decentralized cloud storage to the masses - via more efficient and autonomous payments. + +Here are a few of the benefits of receiving payouts through this approach: + +**Better Scalability** + +With zkSync Era, payouts will consume less block space on the Ethereum network because transactions are bundled together and processed in batches. The system is currently capable of processing 2000 transactions per second! + +**Lower Layer 2 Transfer Fees** + +ZkSync also dramatically lowers network transfer fees (compared to Layer 1 fees) for operators sharing their hard drive space and bandwidth on the Storj network. ZkSync Era accounts are tied to your existing Ethereum keys and current transaction fees on L2 are as low as \~0.000543 ETH at 210 Gwei. As the zkSync Era ecosystem grows, interoperability between projects and exchanges means even more savings. These fees can be reinvested in the community, creating new incentives for network operators to drive growth. + +**Pay Network Fees in STORJ Token** + +One of the most interesting things about zkSync Era is it supports "gasless meta-transactions" that allow users to pay transaction fees in the tokens being transferred. For example, if you want to transfer STORJ from L2 to an exchange, smart contract, or other address, there is no need for you to own ETH or any other tokens. + +[Introducing zkSync Era starting from July 2023!](https://forum.storj.io/t/july-5-2023-ethereum-layer-1-and-zksync-payouts-for-the-month-of-june-are-complete/23167?u=alexey) + +## Get Started and Opt-in + +To opt-in to [zkSync Era](https://zksync.io/) you need to do a simple change in your Node configuration by following these steps: + +## Binary versions (include Windows/Linux GUI) + +Open your storage node's `config.yaml` (see [](docId:gDXZgLlP_rcSW8SuflgqS)) and add/change the line + +```yaml +operator.wallet-features: ['zksync-era'] +``` + +{% callout type="warning" %} +Please enter everything in lowercase and double-check for spelling mistakes. This is a very basic implementation without any validations. +{% /callout %} + +Once you have added/updated the line to your config file, save it and restart your node. + +## Docker versions + +If you use a docker version, you can also specify the `zksync-era` wallet feature as an option after the image name, for example: + +```shell +docker run ... storjlabs/storagenode:latest --operator.wallet-features=zksync-era +``` + +{% callout type="warning" %} +Please enter everything in lowercase and double-check for spelling mistakes. This is a very basic implementation without any validations. +{% /callout %} + +If you decided to specify the `zksync-era` wallet feature as an option, you need to stop and remove the container and run it back with all your parameters include added option for wallet feature, otherwise you can just restart the container. + +## How to check the opt-in for zkSync Era + +1. Navigate to your personal web-dashboard + + You should see an indication of zkSync enabled: + + ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/1dzgaZpKadOoLc2krD5Y1_image.png) + +1. Opt-in for zkSync Era payouts for STORJ payments + +1. Navigate to [zkSync Bridges](https://zksync.io/explore#bridges). + See also [How to add STORJ token to my Wallet](https://forum.storj.io/t/zksync-era-to-transfer-storj-in-binance-wallet/26119/10?u=alexey). + +1. Connect your L1 Ethereum wallet + + If you have problems accessing your wallet, you might want to change your payout address to an address that you can access (for which you control the private keys). + + zkSync Era supported wallets: + + - WalletConnect, an open source protocol for connecting decentralized applications to mobile wallets. + - hardware wallets like Trezor/Ledger + - software wallets like Metamask, MEW, Fortmatic, Portis, Oper, Dapper, Lattice, Torus and many other. + - see also https://docs.zksync.io/build/tooling/wallets.html + +zkSync Era enables our Storage Node Operators to more easily interact directly with the world of DeFi through solutions like ZigZag (zkSync Lite only), Uniswap V3, Binance, ByBit and [others](https://zksync.io/explore#exchanges). + +We are excited to share this update around payment scaling with our community of operators. If you have any questions about using zkSync Era, check out our documentation. + +**If you have ideas, or would like to talk with the team, please feel free to [reach out on our forum](http://forum.storj.io)**. + +You can read more about our approach to storage node payouts in general [here](docId:DVKqtMtnBdZ99gFRWCojP). + +## Understanding zkSync Era fee + +The fee for transaction on zkSync Era (L2 -> L1) can be checked on the [Bridge](https://portal.zksync.io/bridge/withdraw) + +# Transfer tokens from zkSync Era to Ethereum + +1. Navigate to [txSync zkSync Era Bridge](https://app.txsync.io/bridge) + +2. Connect your Etherum Wallet + + ![](https://link.storjshare.io/raw/jvdvvzqjy5sncsehbjh6frkgb46a/docs%2Fimages%2FtxSync-zkSync-Era-WalletConnect.png) + +3. Configure your Wallet to use the zkSync Era Mainnet, following wizard for the chosen Wallet, if you did not already. + +4. Sign the connection + + ![](https://link.storjshare.io/raw/juoh36vchvhrapbh7mhpzj7rc7aa/docs%2Fimages%2FtxSync-zkSync-Era-Sign.png) + +5. Select the STORJ token both for the transfer and the fee, specify an amount and provide a destination Ethereum address + + ![](https://link.storjshare.io/raw/juuomyaaekbsqeuj7eybu5r7wnoq/docs%2Fimages%2FtxSync-zkSync-Era-withdraw.png) + +6. Confirm your selection by button **Continue**. + +7. You will need to sign a transaction in your connected Wallet to transfer your tokens to the provided address. diff --git a/app/page.md b/app/(docs)/page.md similarity index 100% rename from app/page.md rename to app/(docs)/page.md diff --git a/app/support/account-management-billing/_meta.json b/app/(docs)/support/account-management-billing/_meta.json similarity index 100% rename from app/support/account-management-billing/_meta.json rename to app/(docs)/support/account-management-billing/_meta.json diff --git a/app/support/account-management-billing/billing/page.md b/app/(docs)/support/account-management-billing/billing/page.md similarity index 73% rename from app/support/account-management-billing/billing/page.md rename to app/(docs)/support/account-management-billing/billing/page.md index 2a3137f22..05ae84f6f 100644 --- a/app/support/account-management-billing/billing/page.md +++ b/app/(docs)/support/account-management-billing/billing/page.md @@ -21,68 +21,50 @@ Manage your Billing, Invoices and Payment methods The Billing screen allows you to see all your projects and their **Total Estimated Charges** for the current Billing Period and **Available Balance** on the **Overview** tab. You can check **Transactions** for your STORJ deposit address on the **Payment Methods** tab. Your invoices you can see on the **Billing History** tab, your coupons and [](docId:i6OGJ9eZJC7Vw04nKSqcD) you can see and add on the **Coupons** tab. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/3YnX9irXmd-LfzDcs71nn_image.png) - You can expand any Project to see details of the charge. See [](docId:59T_2l7c1rvZVhI8p91VX) for details. ## Add a Payment Method To add a [](docId:7U4_uu6Pzg6u2N6FpV9VE) you can switch to the **Payment Methods** tab and select **Add STORJ Tokens** or Add New Payment Method. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/B72QGAlJzf15QL7tVadUp_image.png) - Please read the [](docId:59T_2l7c1rvZVhI8p91VX) section for details. -## Adding STORJ tokens +### Adding STORJ tokens You can select to **Add STORJ Tokens** on the **_Billing - Payment Methods_** screen, the deposit address will be automatically generated for you. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/SXaLqT-6sp7FMoEBG2mz__image.png) - If you click **Add funds** button, you will see a screen with QR code and your deposit address where you can send your STORJ tokens. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/-xLps8zNN2VHXNIRTNmHh_image.png) - When you will pay the needed amount of STORJ, they will be added automatically to your **Available Balance** in USD value. {% callout type="warning" %} -This deposit address supports only L1 ERC20 STORJ transactions on the Ethereum network. zkSync, zkSync-era. polygon and other Layer 2 protocols are not supported at this time. +This deposit address supports only L1 ERC20 STORJ transactions on the Ethereum network and L2 ERC20 STORJ transactions on the zkSync Era network. zkSync Lite, Polygon and other Layer 2 protocols are not supported at this time. {% /callout %} {% callout type="info" %} Please note, the payment will be accounted only after some amount of confirmations on the Ethereum network and then StorjScan will send them to your balance. This could take from minutes and up to 4 hours. If it took longer, please [contact support](https://supportdcs.storj.io). {% /callout %} -### Viewing transactions +#### Viewing transactions You can click the **See transactions** button in the **Billing - Payment Methods** section to see your transactions: -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/g4vqDj2OU3yjOWwdSra96_image.png) - ### Adding a Card You can select to **Add New Payment Method** to add a Card to your account on the **_Billing - Payment Methods_** screen. You will be prompted to specify Card details. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/C4o1JavxukxpIrcIEGW-B_image.png) - Please provide a valid Card number, expiration date and CVC, then confirm adding a Card with the **Add Credit Card** button. We do not store your card details, they are used to register your Card on [Stripe](https://stripe.com). ## View a Previous Billing Period and Invoices If you select the **Billing History** tab, you will see all previous invoices: -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/1i-OBeqj-5Y48u2ljqStm_image.png) - You can click on **Invoice PDF** link on the right side of the invoice to see details. ## Add Coupons You can see your coupons on the **_Coupons_** tab of the **_Billing_** screen. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/EXnpfXGcvdpypTipra8Ln_image.png) - You can **Apply New Coupon**: -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/k2OfgUDmOO_kMxWyBrcIE_image.png) - The added Coupon will be added as another tile. diff --git a/app/support/account-management-billing/closing-an-account/page.md b/app/(docs)/support/account-management-billing/closing-an-account/page.md similarity index 100% rename from app/support/account-management-billing/closing-an-account/page.md rename to app/(docs)/support/account-management-billing/closing-an-account/page.md diff --git a/app/support/account-management-billing/creating-your-account/page.md b/app/(docs)/support/account-management-billing/creating-your-account/page.md similarity index 52% rename from app/support/account-management-billing/creating-your-account/page.md rename to app/(docs)/support/account-management-billing/creating-your-account/page.md index 068311a1b..f5ca69de4 100644 --- a/app/support/account-management-billing/creating-your-account/page.md +++ b/app/(docs)/support/account-management-billing/creating-your-account/page.md @@ -23,24 +23,6 @@ Learn more about [](docId:v0b3GtAU4dDT_1qibwCxc) under Concepts. To register for an Account, go to Storj.io and choose Start for Free -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/KUG4mPsNpzLXMkulWZJ4W_account01.png) +Next, select your [](docId:v0b3GtAU4dDT_1qibwCxc), fill out the form and sign up. You'll receive an email asking you to verify your email address with a 6-digit one time verification code. -Next, select your [](docId:v0b3GtAU4dDT_1qibwCxc) in this example, the Americas region: - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/E3ie6SDBodo6Xz1t32IeN_account02.png) - -Choose your account type - Personal if you're an individual with a smaller project - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/ProKS3n1_rBPBj3PON-Em_account03.png) - -Or Professional if you are part of a business interested in leveraging Storj in an application or service - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/He52WhNreWbPyINWumw6-_account04.png) - -Fill out the form and sign up. You'll receive an email asking you to verify your email address: - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/by3ZW3r_m_fHXyCGOgml8_account05.png) - -Click confirm to verify your email, then log into the Satellite Admin Console with your username and password. - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/5rma6dFXWcIYqmvAAAitf_account06.png) +Enter the 6-digit code to verify your email, then log into the Storj Console with your username and password. \ No newline at end of file diff --git a/app/support/account-management-billing/data-retention-policy/page.md b/app/(docs)/support/account-management-billing/data-retention-policy/page.md similarity index 100% rename from app/support/account-management-billing/data-retention-policy/page.md rename to app/(docs)/support/account-management-billing/data-retention-policy/page.md diff --git a/app/support/account-management-billing/payment-methods/changing-payment-methods/page.md b/app/(docs)/support/account-management-billing/payment-methods/changing-payment-methods/page.md similarity index 100% rename from app/support/account-management-billing/payment-methods/changing-payment-methods/page.md rename to app/(docs)/support/account-management-billing/payment-methods/changing-payment-methods/page.md diff --git a/app/support/account-management-billing/payment-methods/debits-against-payment-methods/page.md b/app/(docs)/support/account-management-billing/payment-methods/debits-against-payment-methods/page.md similarity index 100% rename from app/support/account-management-billing/payment-methods/debits-against-payment-methods/page.md rename to app/(docs)/support/account-management-billing/payment-methods/debits-against-payment-methods/page.md diff --git a/app/support/account-management-billing/payment-methods/deleting-a-payment-method/page.md b/app/(docs)/support/account-management-billing/payment-methods/deleting-a-payment-method/page.md similarity index 100% rename from app/support/account-management-billing/payment-methods/deleting-a-payment-method/page.md rename to app/(docs)/support/account-management-billing/payment-methods/deleting-a-payment-method/page.md diff --git a/app/support/account-management-billing/payment-methods/expired-credit-card/page.md b/app/(docs)/support/account-management-billing/payment-methods/expired-credit-card/page.md similarity index 100% rename from app/support/account-management-billing/payment-methods/expired-credit-card/page.md rename to app/(docs)/support/account-management-billing/payment-methods/expired-credit-card/page.md diff --git a/app/support/account-management-billing/payment-methods/page.md b/app/(docs)/support/account-management-billing/payment-methods/page.md similarity index 95% rename from app/support/account-management-billing/payment-methods/page.md rename to app/(docs)/support/account-management-billing/payment-methods/page.md index acb9fbc5c..70a073761 100644 --- a/app/support/account-management-billing/payment-methods/page.md +++ b/app/(docs)/support/account-management-billing/payment-methods/page.md @@ -27,9 +27,7 @@ Begin by selecting "Billiing" from the "My Account" dropdown menu at the left bo ## Using a Credit Card -You can select to **Add New Payment Method** to add a Card to your account on the **_Billing - Payment Methods_** screen. You will be prompted to specify Card details. - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/C4o1JavxukxpIrcIEGW-B_image.png) +You can select to **Add New Card** to add a Card to your account on the **_Billing - Payment Methods_** screen. You will be prompted to specify Card details. You’ll be prompted to add your card information. Using a credit card is somewhat self-explanatory, but there are some key points users should understand: diff --git a/app/support/account-management-billing/payment-methods/promotional-credits/page.md b/app/(docs)/support/account-management-billing/payment-methods/promotional-credits/page.md similarity index 100% rename from app/support/account-management-billing/payment-methods/promotional-credits/page.md rename to app/(docs)/support/account-management-billing/payment-methods/promotional-credits/page.md diff --git a/app/support/account-management-billing/reporting-a-payment-problem/page.md b/app/(docs)/support/account-management-billing/reporting-a-payment-problem/page.md similarity index 100% rename from app/support/account-management-billing/reporting-a-payment-problem/page.md rename to app/(docs)/support/account-management-billing/reporting-a-payment-problem/page.md diff --git a/app/support/account-management-billing/requesting-a-refund/page.md b/app/(docs)/support/account-management-billing/requesting-a-refund/page.md similarity index 100% rename from app/support/account-management-billing/requesting-a-refund/page.md rename to app/(docs)/support/account-management-billing/requesting-a-refund/page.md diff --git a/app/support/dashboard/page.md b/app/(docs)/support/dashboard/page.md similarity index 96% rename from app/support/dashboard/page.md rename to app/(docs)/support/dashboard/page.md index 99663f65a..90a5c66d3 100644 --- a/app/support/dashboard/page.md +++ b/app/(docs)/support/dashboard/page.md @@ -23,8 +23,6 @@ Learn more about Projects in [](docId:M-5oxBinC6J1D-qSNjKYS) under Concepts. On the Project Dashboard, there are a number of navigational elements and information displays: -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/ciTyxd2Es4DaozorgD9wQ_dashboardpro1-projectactive.png) - 1. **Projects management** - This element allows you to add Projects and switch between different Projects. There you also have a [](docId:jwCUqpDCk8CUuUqFuykFx) setting. 2. **Project Navigation** - This element allows you to move between the different functions related to the project you have selected, to view the [](docId:k6QwBZM3hnzxkCuQxLOal), use the [](docId:4oDAezF-FcfPr0WPl7knd) to interact with data stored on Storj through a web browser interface, create [](docId:XKib9SzjtEXTXWvdyYWX6) for native integrations and credentials for the [](docId:yYCzPT8HHcbEZZMvfoCFa), invite other developers to collaborate with you on your project in [](docId:0_4hY4Dp5ju9B8Ec6OTf3), see [](docId:Hurx0SirlRp_O5aUzew7_), and **Quick Start**, and manage your Account in **My Account**. diff --git a/app/support/faqs/page.md b/app/(docs)/support/faqs/page.md similarity index 100% rename from app/support/faqs/page.md rename to app/(docs)/support/faqs/page.md diff --git a/app/support/object-browser/page.md b/app/(docs)/support/object-browser/page.md similarity index 59% rename from app/support/object-browser/page.md rename to app/(docs)/support/object-browser/page.md index d824a6859..50582709c 100644 --- a/app/support/object-browser/page.md +++ b/app/(docs)/support/object-browser/page.md @@ -22,7 +22,7 @@ By using hosted Gateway MT you are opting into **server-side encryption**. See [ ## Configure Object Browser Access -**Navigate to the** [](docId:pxdnqsVDjCLZgeEXt2S6x) page within your project. If you do not have any buckets yet - we will create a `demo-bucket` for you. +**Navigate to the** [](docId:pxdnqsVDjCLZgeEXt2S6x) page within your project. When you click on the bucket, you will be prompted to read carefully - The object browser uses [](docId:hf2uumViqYvS1oq8TYbeW). @@ -30,20 +30,12 @@ When you click on the bucket, you will be prompted to read carefully - The objec Don't forget to save your **Encryption Passphrase** generated below, you will need it for future access. {% /callout %} -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/PgEXOy3cK2ue1zGwGqxdh_qsobject01.png) - If this is your first time using the object browser, you **must create an encryption passphrase.** We strongly encourage you to use a mnemonic phrase. The GUI automatically generates one on the client side for you with the **Generate passphrase** option. You can also download it as a text file. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/SWYh6j1RWfLrc4dPlgYW2_qsobject02.png) - Alternatively, you can enter your own passphrase using the **Enter passphrase** option. Finish selection by click on **Continue** button. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/m_4pzkqRUSiGpOXmnWd60_qsobject03.png) - To continue, you need to mark the checkbox **_\[v] I understand, and I have saved the passphrase._** This will enable the **Continue** button. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/VJondlvfOjDcc04ILsAsF_qsobject04.png) - When you click the **Continue** button, you will be placed into the **Objects** view if you already have buckets, otherwise a new bucket **_demo-bucket_** will be created and you will be placed into that bucket view. ## Upload files and folders @@ -56,8 +48,6 @@ To upload larger files, please utilize the [](docId:TbMdOGCAXNWyPpQmH6EOq). If you have not yet created a bucket, the bucket **_demo-bucket_** will be created automatically to allow you to upload objects right away. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/A1VBtbjhSjxV187WZEiAH_qsobject04.png) - To upload your first object, **drag it into the browser** or select **Upload File** and browse to the file you wish to upload. You can upload not only files but also folders, just **drag them into the browser** or select **Upload Folder** and browse to the folder you wish to upload. @@ -65,36 +55,28 @@ You can upload not only files but also folders, just **drag them into the browse If you want to create a folder, you can do that with the **New Folder** button. {% callout type="success" %} -When you drag and drop your file into the Satellite Admin Console Object Browser, the Storj S3-compatible Gateway will encrypt the data using [](docId:hf2uumViqYvS1oq8TYbeW), break large files into 64MB Segments (or for smaller files a single segment), then erasure code the segments, breaking each segment into 80 pieces, then distributing those pieces over our network of thousands of independently operated storage nodes. +When you drag and drop your file into the Storj console, the Storj S3-compatible Gateway will encrypt the data using [](docId:hf2uumViqYvS1oq8TYbeW), break large files into 64MB Segments (or for smaller files a single segment), then erasure code the segments, breaking each segment into 80 pieces, then distributing those pieces over our network of thousands of independently operated storage nodes. {% /callout %} ## Deleting files 1. If you select the three vertical dots on the right side of a file, a popup menu will appear: - ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/pEw5qFRbraYchiz0mtOv7_qsobject05.png) - 2. Select the **Delete** command. - ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/JRQ09me42z8yIy05hLLn0_qsobject06.png) - -3. Confirm deletion with **Yes**. +3. Confirm deletion with **Delete**. ## Creating buckets Buckets are your containers that store objects. -You can create your buckets in the **Objects** view or if you click on the **<-Back to Buckets** button, in the bucket view. - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/oJ74hmgmN9h5iDemALwMk_qsobject07.png) +You can create your buckets in the **Browse** view or if you click on the **<-Back to Buckets** button, in the bucket view. {% callout type="warning" %} The bucket name can only contain lowercase letters, numbers, and hyphens. {% /callout %} -To create a new bucket, click the **New bucket** button in the **Buckets** view. A new module window will pop up called **Create Bucket**. Please provide a name using only lower case alphanumeric characters and dashes (this is a limitation for compatibility with existing object storages). - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/Yewew2V1tdS66o93P_XIM_qsobject08.png) +To create a new bucket, click the **New bucket** button in the **Browse** view. A new module window will pop up called **Create Bucket**. Please provide a name using only lower case alphanumeric characters and dashes (this is a limitation for compatibility with existing object storages). After creating your new bucket, you will be placed into the bucket where you can [](docId:gh5RtIDbMkAoomljO7f8d) @@ -102,13 +84,9 @@ After creating your new bucket, you will be placed into the bucket where you can 1. Clicking the three vertical dots on the right side of the bucket, a popup menu will appear: - ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/5YJzM4uQlX2DkoFIuyNp1_qsobject09.png) - 2. Click the **Delete** command - ![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/awroxWO45F35KG6mW6zys_qsobject10.png) - -3. Type the **_Bucket Name_** and **Confirm Delete Bucket**. +3. Type the **_Bucket Name_** and **Delete Bucket**. {% callout type="warning" %} Be careful when deleting buckets - If you still have objects in the bucket being deleted, they will be deleted too! @@ -118,30 +96,18 @@ Be careful when deleting buckets - If you still have objects in the bucket being After an upload completes, you will have the option of creating a share link. If you wish, click the file name - it will open a preview with a map. Here you can click the **Share** button. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/vCYGlW3Gy5TJHiQHzLT_n_qsobjectshare01.png) - Or you can click on the three vertical dots to the right of the file you want to share, and select **Share** to share your object. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/AvXeXUPRK2-PFiOyCGS6Z_qsobjectshare02.png) - The **Share** pop-up window allows you to share the link via social media or copy it with **Copy Link**. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/GmA5kH_adiXmGOaw7iTAz_qsobjectshare03.png) - The share link includes a rendering of where the pieces of your file are located on the globally distributed network of storage nodes, as well as a preview of that file. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/y1Z-utzw4fEsvj6gffynu_qsobjectshare04.png) - ## Share a bucket 1. Click the three vertical dots to the right of the bucket, a popup menu will appear - ![Share Bucket](https://link.storjshare.io/raw/juc6cqmbtg2qgaja6jzx67zfkjwa/docs/images/Share-Bucket.png) - 2. Click the **Share Bucket** command - ![Share the link](https://link.storjshare.io/raw/jwkhjjn4le4udilzxbeoe3wj7iwa/docs/images/Share%20a%20link.png) - -3. Click the needed button to share the link +3. Click the copy button to copy and share the link This concludes the Object Browser Quickstart. diff --git a/app/support/page.md b/app/(docs)/support/page.md similarity index 100% rename from app/support/page.md rename to app/(docs)/support/page.md diff --git a/app/support/projects/page.md b/app/(docs)/support/projects/page.md similarity index 74% rename from app/support/projects/page.md rename to app/(docs)/support/projects/page.md index f76bf596d..9f9d19d37 100644 --- a/app/support/projects/page.md +++ b/app/(docs)/support/projects/page.md @@ -24,8 +24,6 @@ Learn more about Projects in [](docId:M-5oxBinC6J1D-qSNjKYS) under Concepts. To select, create or **Manage Projects** you can click the name of your project on the left side toolbar above Dashboard. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/5jcrdDKiEwLzjuqCYqOPB_projects1.png) - ## Create a new Project On **_Projects_** screen to create a new Project select the **Create Project**. On Project **Dashboard** you can click the name of the current project and select **Create Project**. @@ -34,24 +32,18 @@ On **_Projects_** screen to create a new Project select the **Create Project**. The availability of this function depends on your account tier. Please check [](docId:Zrbz4XYhIOm99hhRShWHg) for details. {% /callout %} -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/_75DWodmOKqwaDRytJXvN_projects2.png) - Specify the **Project Name**, optional **Description** and confirm the creating with the **Create Project** button. ## Modify the existing Project To modify the existing Project on the **_Projects_** screen you can select a needed project and modify its name or description. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/5GPeq8Gd2lQ6eE28f3f8X_projects3.png) - ## Changing Project Limits If your account tier allows you to change your [](docId:Zrbz4XYhIOm99hhRShWHg), you will have more options than a free plan. Select **Edit** to the right of the limit to change it. However, it will not allow to increase limits greater than your available maximum. To change the maximum you need to file a support request to change your limits, see [](docId:A4kUGYhfgGbVhlQ2ZHXVS). -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/Uw86kTuJHbXNQCOZ1lGba_projects5.png) - ## Delete the existing Project At the moment the Satellite Admin Console will not allow you to delete a Project. @@ -70,27 +62,18 @@ We do not have an access to your data and Access Grants, because they are encryp You may manage your [Passphrase](docId:M-5oxBinC6J1D-qSNjKYS#encryption-key) used for [Buckets](docId:4oDAezF-FcfPr0WPl7knd) view. -![](https://link.storjshare.io/raw/jvqobk4svumlsdxgn66o4heyi75q/docs%2Fimages%2FManage%20Passphrase%202023-12-28%20102824.png) - The **Manage Passphrase** window will allow you to: * [Create a new Passphrase](#create-a-new-passphrase) * [Switch current Passphrase](#switch-the-current-passphrase) * [Clear saved Passphrase](#clear-the-saved-passphrase) -![](https://link.storjshare.io/raw/ju5pajakxkq6iykj33fueoyzwjya/docs%2Fimages%2FManage%20Passphrase%20-%20choice%202023-12-28%20103213.png) - ### Create a new Passphrase -![](https://link.storjshare.io/raw/jvodweijki7eso7z2u4rd2oawika/docs%2Fimages%2FCreate%20a%20new%20passphrase%202023-12-28%20103850.png) - After click the **Continue** button you will have a choice how do you want to provide your Encryption Passphrase: * [Generate a 12-words Encryption Passphrase](#generate-12-word-passphrase) * [Enter your own Encryption Passphrase](#enter-a-new-passphrase) -![](https://link.storjshare.io/raw/jwutrovcovetzu3lhi7j2wxgfgxq/docs%2Fimages%2FEncryption%20Passphrase%202023-12-28%20104218.png) - #### Generate 12-word passphrase -![](https://link.storjshare.io/raw/jx57khok3xb4t52trdjmjhhrammq/docs%2Fimages%2FPassphrase%20Generated%202023-12-28%20104455.png) Now you may: * **Copy** the generated Passphrase @@ -100,16 +83,13 @@ Now you may: You need to select a checkbox **[ ] Yes, I saved my encryption passphrase** to **Continue** #### Enter a new passphrase -![](https://link.storjshare.io/raw/jw34uudu6bmdzycjed7vr726rxzq/docs%2Fimages%2FEnter%20Passphrase%202023-12-28%20105653.png) You should enter your own Encryption Passphrase (or a previously used), select a checkbox **[ ] Yes, I saved my encryption passphrase** to **Continue** ### Switch the current Passphrase -![](https://link.storjshare.io/raw/jubisqg4sjzbk7t7nx3w4bzjavya/docs%2Fimages%2FSwitch%20Passphrase%202023-12-28%20110317.png) You should enter your own Encryption Passphrase (or a previously used) to **Continue** ### Clear the saved Passphrase -![](https://link.storjshare.io/raw/jwzhuxi7onvp7jc7cfgdipnlmqra/docs%2Fimages%2FClear%20my%20passphrase%202023-12-28%20110818.png) Click the **Continue** button to clear your currently saved Encryption Passphrase diff --git a/app/support/storj-console/page.md b/app/(docs)/support/storj-console/page.md similarity index 100% rename from app/support/storj-console/page.md rename to app/(docs)/support/storj-console/page.md diff --git a/app/support/usage-limit-increases/page.md b/app/(docs)/support/usage-limit-increases/page.md similarity index 100% rename from app/support/usage-limit-increases/page.md rename to app/(docs)/support/usage-limit-increases/page.md diff --git a/app/support/users/page.md b/app/(docs)/support/users/page.md similarity index 76% rename from app/support/users/page.md rename to app/(docs)/support/users/page.md index a4db0403c..028faf515 100644 --- a/app/support/users/page.md +++ b/app/(docs)/support/users/page.md @@ -21,12 +21,8 @@ If you need to collaborate with other developers on a project, you can add other When you add another user to your project, that user will have full access to the Project Dashboard, Object Browser, and access Grants for your Project. {% /callout %} -Navigate to the **Users** screen. +Navigate to the **Team** screen. -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/eMttBc7nDmSUgP9Y-OAnI_users1.png) - -Select the **Add** button - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/Uv5sm1Bh3hC5SPbinJvIm_users2.png) +Select the **Add Member** button Type in the email addresses that the users have registered with their Satellite Accounts. The Users will be added to the Project Team and notified via email. diff --git a/app/layout.js b/app/layout.js index beee82b4a..c8ca06993 100644 --- a/app/layout.js +++ b/app/layout.js @@ -1,12 +1,9 @@ import { Inter } from 'next/font/google' -import localFont from 'next/font/local' import clsx from 'clsx' +import Script from 'next/script' import { ThemeProvider } from '@/components/theme-provider' -import { Navigation } from '@/components/Navigation' import Navbar from '@/components/Navbar' -import { Hero as HeroWrap } from '@/components/Hero.client' -import { Hero } from '@/components/Hero' import '@/styles/tailwind.css' @@ -40,23 +37,19 @@ export default function RootLayout({ suppressHydrationWarning > + {process.env.NODE_ENV === 'production' && ( +