219 links
Rustで"Hello World"を出力できる小さなWasm Runtimeをゼロから実装して、WasmとWASIの動作原理を理解する本です。
Implementing a Wasm runtime in Rust
Swarm Testing paper
Zig File I/O (
std.Io)
This course is on the design and implementation of database management systems. Topics include data models (relational, document, key/value), storage models (n-ary, decomposition), query languages (SQL, stored procedures), storage architectures (heaps, log-structured), indexing (order preserving trees, hash tables), transaction processing (ACID, concurrency control), recovery (logging, checkpoints), query processing (joins, sorting, aggregation, optimization), and parallel architectures (multi-core, distributed). Case studies on open-source and commercial database systems are used to illustrate these techniques and trade-offs. The course is appropriate for students that are prepared to flex their strong systems programming skills.
Phil Eaton's blog
Scheduling multithreaded computations by work stealing
Introducing kv, a statically allocated key/value server.
Static Allocation with Zig - Introducting kv, a statically allocated key/value server
🔔 FOLLOW GOLAB CHANNEL 🔔ABSTRACT:The Go runtime is an essential part of every single Go compiled binary. Understanding what it does for us and the price w...
The financial transactions database designed for mission critical safety and performance. - tigerbeetle/tigerbeetle
Tigerbeetle VSR
https://cppcon.org/---Reflection: C++’s Decade-Defining Rocket Engine - Herb Sutter - CppCon 2025In June 2025, C++ crossed a Rubicon: it handed us the keys t...
Reflection: C++’s Decade-Defining Rocket Engine - Herb Sutter - CppCon 2025
Writing a sqlite clone from scratch in C
SQLite File Format Viewer
Write-Ahead Logging
A customer query was taking over 4× longer than it should have. The profiler pointed at Rust code we'd assumed was free. We followed the trail all the way down to assembly to find the true cost.
Rust zero-cost abstraction vs SIMDI
vu128: Efficient variable-length integers
Polymorphic, Recursive Interfaces Using Go Generics
Short presentation by Alex Gaetano Padula for Bengaluru Systems.
TidesDB Bengaluru Systems Presentation
Darwins deceptive durability
Legacy mirror of Darwin Kernel. Replaced by https://github.com/apple-oss-distributions/xnu - apple/darwin-xnu
darwin-xnu/bsd/vfs/vfs_cluster.ccomment onMIN_DIRECT_WRITE_SIZEset to 16384 governing how much I/O should be allowed before considering to allow the caller to bypass the buffer cache, not allowing I/O less than 16k to bypass the UBC
This specification defines a Uniform Resource Name namespace for UUIDs (Universally Unique IDentifier), also known as GUIDs (Globally Unique IDentifier). A UUID is 128 bits long, and can guarantee uniqueness across space and time. UUIDs were originally used in the Apollo Network Computing System and later in the Open Software Foundation\'s (OSF) Distributed Computing Environment (DCE), and then in Microsoft Windows platforms. This specification is derived from the DCE specification with the kind permission of the OSF (now known as The Open Group). Information from earlier versions of the DCE specification have been incorporated into this document. [STANDARDS-TRACK]
RFC 4122: A Universal Unique Identification (UUID) URN Namespace
This work addresses the complexities involved in designing distributed quantum algorithms, highlighting that quantum entanglement does not bypass the Fischer-Lynch-Paterson (FLP) impossibility theorem in asynchronous networks. Although quantum resources such as entanglement offer potential speedups, the inherent constraints of classical communication remain. We develop a leader election algorithm as a proof of concept, demonstrating how entanglement can enhance efficiency while still contending with asynchronous delays. This algorithm serves as a foundation for a broader blueprint for future distributed quantum algorithms, providing insights into both the real performance gains and the limitations that entanglement offers in a distributed setting.
Pushing boundaries: Quantum-Enhanced Leader Election and Limits of Consensus
An introduction to CLX Interconnect
Every programming language environment has an interface for streaming data. Using real world examples of compression, files, sockets, and network protocols, ...
An Illustrated Proof of the CAP Theorem
- Linux, Oracle, SQL performance tuning and troubleshooting training & writing.
Tanel Poder Blog
The financial transactions database designed for mission critical safety and performance. - tigerbeetle/tigerbeetle
Tigerbeetle DST and VOPR
Ceph is a distributed object, block, and file storage platform - ceph/ceph
The Lemon LALR(1) Parser Generator
Yes, this is the premier course at Carnegie Mellon University on the design and implementation of database management systems. Topics include data models (relational, document, key/value), storage models (n-ary, decomposition), query languages (SQL, stored procedures), storage architectures (heaps, log-structured), indexes + filters (order preserving trees, hash tables, vector indexes), transaction processing (ACID, concurrency control), recovery (logging, checkpoints), query processing (joins, sorting, aggregation, optimization), and parallel architectures (multi-core, distributed). Case studies on open-source and commercial database systems are used to illustrate these techniques and trade-offs. The course is appropriate for students that are prepared to sweat their systems programming skills.
CMU Andy Pavlo
Posted by Joshua Bloch, Software EngineerI remember vividly Jon Bentley's first Algorithms lecture at CMU, where he asked all of us incoming Ph.D. ...
Nearly all Binary Searches and Mergesorts are broken
WiredTiger's source tree. Contribute to wiredtiger/wiredtiger development by creating an account on GitHub.
Last week, we saw the difference between Cooperative and Preemptive scheduling and how it enables resources-efficient I/O operations. Today, we are going to learn how runtimes work under the hood. This post contains excerpts from my book Black Hat Rust Rust does not provide the execution context required to execute
A guide to writing a DNS Server from scratch in Rust - EmilHernvall/dnsguide
dns in rust -
std::pin, what is it
It’s been nine years since Designing Data-Intensive Applications became the standard text for distributed systems. Today, Martin Kleppmann and Chris Riccomin...
Re-Designing Data-Intensive Applications: The Shift to Cloud-Native Storage
http://db.cs.cmu.edu/seminar2015/ Embedded databases: They're the boxer briefs of the database world in that they are underneath a wide variety of applicatio...
CMU
sqlite3 db file parser. Contribute to bwaklog/parsley.py development by creating an account on GitHub.
A sqlite basic file parser
Learn what a unikernel is and how it compares to a Linux container. Build and run your first unikernel inside a playground box.
Hands-On Introduction to Unikernels
Go uses goroutines while languages like Java use threads.
This post is part of my Summer Autumn of Rust 2024 series: weekly (or more) quickies about how to write great Rust code so you will be ready to rock at the end of the summer and find a new exciting job 🤘 Dont forget to Subscribe (Email & RSS)
Please stop calling databases CP or AP
Web Browser Engineering by Pavel Panchekha & Chris Harrelson
Async programming in Rust is built on top of the operating system's async IO facilities. While it is possible to just use async/await for control flow, mostly people use async for async IO. The runtime and libraries abstract away many of the details of IO (which is good, because
Asynchronous IO fundamentals
MPEG-2 Transport Stream Packaging for Media Over QUIC Transport
Alex Rebert, Software Engineer, Christoph Kern, Principal Engineer, Security Foundations Google’s Project Zero reports that memory safety v...
Google's Perspective on Memory Safety
I feel like this is too long! what details do you think would be better in a different post? or do you think the focus of this one should change? perhaps this should not go into any details and just link to other posts for e.g async/concurrency? With 0.16 just released with the std.Io interface. It is bound to change (this will be updated with it) in the future as more people use it and find where it is lacking, many areas are already known. It is only in a “good enough to be used for most ...
std.Iooverview
Channels provide a simple mechanism for goroutines to communicate, and a powerful construct to build sophisticated concurrency patterns. We will delve into t...
Foundation DB technical overview
In this blog post, we walk through several different implementations of key-value separation in LSM storage engines.
Threads were designed to parallelize compute-intensive tasks. However, these days, a lot of applications (such as a network scanner) are I/O (Input / Output) intensive. Thus, threads have two significant problems: They use a lot of memory (compared to others solutions). Launches and context switches have a cost that can
Will Wilson, Co-founder & CEO at Antithesis (https://antithesis.com/) presents a detailed exploration of swarm testing, a software testing technique that fin...
Will Wilson on Swarm Testing -- Papers We Love SF March 2026
The Impact of Thread-Per-Core Architecture on Application Tail Latency
Amazon Aurora is a relational database service for OLTP workloads offered as part of Amazon Web Services (AWS). In this paper, we describe the architecture of Aurora and the design considerations leading to that architecture. We believe the central constraint in high throughput data processing has…
Amazon Aurora: Design considerations for high throughput cloud-native relational databases
A library that provides an embeddable, persistent key-value store for fast storage. - facebook/rocksdb
Rocksdb MemTable
The AT Protocol (Authenticated Transfer Protocol, or atproto) is a standard for public conversation and an open-source framework for building social apps.
The AT Protocol
Vectored I/O (wikipedia) for scatter gather operations, with
sys/uio.h,readv,writev
arxiv/catchup
This is probably already well known, but it bit me while planning to do some comparative benchmarks between Linux and OSX. I started with OSX and direct=1, rw=randread, but noticed that OSX was rea...
(got above from this) Issue: OSX
fcntl(fd, f_NOCACHE, 1)is not the same asO_DIRECTon Linux
(educational) build your own disk based KV store. Contribute to avinassh/py-caskdb development by creating an account on GitHub.
Pekka Enberg's blog
Lessons learned building TidesDB - an embeddable storage engine that outperforms RocksDB through lock-free concurrency, adaptive compaction, and aggressive caching.
What I Learned Building a Storage Engine That Outperforms RocksDB
Zig book
We're building libraries and tools in the Rust programming language! The streams are intended for users who are already somewhat familiar with Rust, but who want to see something larger and more involved be built. You can see earlier related videos by looking at the playlists and uploads on this channel. I post about upcoming streams and ideas for new ones at https://discord.jonhoo.eu, as well as on Twitter, Mastodon, and LinkedIn! Q&A tends to happen using https://wewerewondering.com/. You can sponsor my work at https://github.com/sponsors/jonhoo/. Streaming schedule: https://calendar.google.com/calendar/embed?src=719b6cf08a9f11da4a732083a43aa00a61a7943ea864a57417b5da13b1fc8ccb%40group.calendar.google.com&ctz=Europe%2FOslo The hope of this channel is to help serve the Rust 2018 Roadmap goal of better serving intermediate Rustaceans: https://blog.rust-lang.org/2018/03/12/roadmap.html#better-serving-intermediate-rustaceans Also known as (for search purposes): jonhoo
Thread-per core
Ted Nyman - High Performance Git
An example including both algorithms, where swapping their places clearly does the wrong thing.
How Paxos and Two-Phase Commit Differ
Compute Express Link (CLX)
This is the story of how we scaled our vector search infrastructure by 10x while simultaneously reducing costs by 90 percent over the past two years.
Two years of vector search at Notion: 10x scale, 1/10th cost
Filesystems, file IO, and durability.
Userland Disk I/O
Linux: When to use scatter/gather IO (readv,writev) vs large buffer with fread
Reading Sqlite Schema Tables the Hard Way
Explore SQLite Internals: How The World's Most Used Database Works by Abdur-Rahmaan Janhangeer. Deep dive into database internals, systems engineering, and advanced Python.
SQLite Internals: How The World's Most Used Database Works
@rodio Reading varints conceptually works like this: Extract the lowest 7 bits from every byte Concat the 7-bit groups Interpret the concatenated binary number In this case: 0x81 == 1000 0001 (The leading 1 means continue reading.) extract: 000 0001 0x47 == 0100 0111 (The leading 0 means stop reading.) extract: 100 0111 Concat 000 0001 + 100 0111 == 000 0001 100 0111 == 000 000 1100 0111 == 1100 0111 (This is 199 in decimal.)
SQLite Varint Decoding Stage SZ4
IOP Systems blogs
Modern C++ Tutorial book
Leaarncpp
Cornus: Atomic Commit for a Cloud DBMS with Storage Disaggregation
Awesome materials about database development. Contribute to huachaohuang/awesome-dbdev development by creating an account on GitHub.
The Cockroach Labs blog is where you will hear from the folks bringing CockroachDB to life. We're opinionated. We're irreverent. We're committed to honesty.
Difference between "read commited" and "repeatable read" in SQL Server
Foundation DB - Simulation and Testing
Chaitanya Bhandari from TigerBeetle (https://tigerbeetle.com/) presents TigerBeetle’s approach to testing distributed database systems using Protocol Aware D...
Protocol-aware deterministic simulation testing | Chaitanya Bhandari | Bug Bash 2026
Jamie Brandon blog
Some bugs are so rare, they can take years to track down and fix. What if you could find and fix five of them per day? For Joran Dirk Greef, creator of the T...
Fixing five "two-year" bugs per day
Paper
This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm.
QUIC: A UDP-Based Multiplexed and Secure Transport
Distributed Computing Musings
CacheSack: Admission Optimization for Google Datacenter Flash Caches
Async IO on Linux: select, poll, epoll
Compio with executor and dispatcher decoupled that can be swapped
Last fall I was working on a library to make a safe API for driving futures on top of an an io-uring instance. Though I released bindings to liburing called iou, the futures integration, called ostkreuz, was never released. I don’t know if I will pick this work up again in the future but several different people have started writing other libraries with similar goals, so I wanted to write up some notes on what I learned working with io-uring and Rust’s futures model.
Notes on io_uring
Understanding LSTM Networks
Redpanda Engineering Blog
Notes on epoll and io_uring - OS readiness vs OS completion models
Rustで変数を終了ステータスとして利用する実行バイナリを生成できる小さなリンカーをゼロから実装して、リンカーの動作原理を理解する本です。
Implementing a small linker with Rust
Bluesky is a new social network built upon the AT Protocol, a decentralized foundation for public social media. It was launched in private beta in February 2023, and has grown to over 10 million registered users by October 2024. In this paper we introduce the architecture of Bluesky and the AT Protocol, and explain how the technical design of Bluesky is informed by our goals: to enable decentralization by having multiple interoperable providers for every part of the system; to make it easy for users to switch providers; to give users agency over the content they see; and to provide a simple user experience that does not burden users with complexity arising from the system's decentralized nature. The system's openness allows anybody to contribute to content moderation and community management, and we invite the research community to use Bluesky as a dataset and testing ground for new approaches in social media moderation.
Bluesky and the AT Protocol: Usable Decentralized Social Media
Files are hard (consistency)
Modern C++ Programming Course (C++03/11/14/17/20/23/26)
Modern CPP Programming
This article explains the differences between goroutines and threads and how goroutines work in Go programming language.
Lockless algorithms are of interest for the Linux kernel when traditional locking primitives ei [...]
Introduction to lockless programming
Asynchronous I/O programming with
io_uring
AT Protocol is the tech developed at Bluesky for open social networking. In this article we're going to explore atproto from the perspective of distributed backend engineering.
Atproto for distributed systems engineers
FastCGI: 30 Year Old and Still the Better Protocol for Reverse Proxies
Data Replication Design Spectrum
Amazon S3 is one of the largest distributed systems ever built, storing and serving data for a significant portion of the internet. Behind its simple interfa...
How AWS S3 is built
Learn how the Google Colossus distributed storage system determines how to place files on HDD vs. SSD to balance cost and performance.
Colossus under the hood: How we deliver SSD performance at HDD prices
A thread-per-core async Rust runtime with IOCP/io_uring/polling. - compio-rs/compio
Compio - a thread-per-core async runtime with io_uring
Building a serverless ACID database with one neat trick (atomic PutIfAbsent)
Dive deeper into the concepts behind the AT Protocol.
Understanding Atproto
A library that provides an embeddable, persistent key-value store for fast storage. - facebook/rocksdb
Rocksdb Overview
Simon Willison’s Weblog
How S2 uses deterministic simulation testing for async Rust to find distributed-systems bugs before they reach production.
Deterministic simulation testing for async Rust
Magma: A High Data Density Storage Engine Used in Couchbase
Keynote by Martin Kleppmann at the 15th ACM International Conference on Distributed and Event-based Systems (ACM DEBS 2021)Paper: https://dl.acm.org/doi/10.1...
Martin Kleppman Thinking in Events: from databases to distributed collaboration
How does one do Raw IO on Mac OS X? (ie. equivalent to Linux's O_DIRECT flag)
In my distributed systems course a couple weeks ago, we were discussing how the anti-entropy mechanism in the Dynamo key-value store uses Merkle trees to efficiently compare the data stored on different replicas. The idea is this: suppose that two replicas (on different physical machines, perhaps on opposite sides of the planet) are each storing some large number of key-value pairs. Suppose you want to make sure that the values of the keys on replica A are the same as those of the corresponding keys on replica B. (They might not agree if, say, replica A got updated at a time when replica B was unreachable.)
How I learned about Merklix trees (without having to become a cryptocurrency enthusiast)
on cooperative scheduling in rust async runtime
Course schedule: https://www.cs.uic.edu/~ckanich/cs361/f20/schedule.html
Understanding mmap, the workhorse behind keeping memory access efficient in linux
Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed - apache/iggy
Apache Iggy: Hyper efficient Message Streaming
Seiya Nuta's blog
Enjoy the videos and music that you love, upload original content and share it all with friends, family and the world on YouTube.
video by matklad @ tigerbeetle
Hosting SQLite databases on Github Pages
Some folks who read TIGER_STYLE.md (the TigerBeetle coding guide) are surprised to learn that TigerBeetle allocates no memory after startup.
A Database Without Dynamic Memory Allocation
WiscKey: Separating Keys from Values in SSD-conscious Storage
I/O access and interrupts
To BLOB or Not to BLOB : Large Object Storage in a Database or a Filesystem
The Rollback Journal
Explore database durability, the "D" in ACID. Learn how databases ensure data survives failures & the role of transaction logs.
Decoding Durability - The D in ACID
BPF-DB: A Kernel-Embedded Transactional Database Management System For eBPF Applications
Pushdown Automaton
Educational blog posts for Rust beginners. Contribute to pretzelhammer/rust-blog development by creating an account on GitHub.
Observations across the software engineering industry.
The Pragmatic Engineer by Gergely Orosz
Phil Eaton blogs tagged databases
The transactional blog by Alex Miller
Lecture 2: RPC and ThreadsMIT 6.824: Distributed Systems (Spring 2020)https://pdos.csail.mit.edu/6.824/
MIT 6.824 Lecture 2: RPC and Threads
Async Runtimes slides by Sanchayan Maity
Different I/O Access Methods for Linux, What We chose for ScyllaDB, and why
A minimal distributed key-value database with Hashicorp's Raft library
In the previous article , we explored FAT32—a filesystem that conquered the world through simplicity. Its linked-list approach gets the job done, but it comes with real limitations: no crash protection, linear directory searches that slow down as directories grow, and a hard 4GB ceiling on file sizes. Ext4 takes a fundamentally different approach. Where FAT32 uses a single table as both its allocation tracker and file map, ext4 divides the disk into block groups, each with its own dedicated structures for tracking free space and storing file metadata. Where FAT32 walks a linked list to find your data, ext4 uses extents—a compact B-tree where each entry describes a contiguous run of blocks. And where FAT32 has no crash protection at all, ext4 uses journaling to ensure your filesystem stays consistent after unexpected shutdowns.
Filesystems: Ext4
(Note, this is a variation on this article, invited into docs with the release of 0.16.) As always, reply/comment all you want, and I’ll endeavor to make changes accordingly. Motivation, and migration Zig 0.16 brought big changes to I/O. std.fs references deprecate everywhere, and understanding basic use of std.Io is in order. Zig’s new I/O model brings I/O into alignment with the Allocator model - all of Zig’s “file system, networking, timers, synchronization, and pretty much everything that ...
File I/O basics (0.16)
Build Your Own Database From Scratch in Go - From B+Tree To SQL
Build Your Own Database From Scratch in Go
Database File Format
Understand database consistency (the "C" in ACID) and its importance. Learn how databases maintain data correctness through rules and constraints.
Decoding Consistency - The C in ACID
What I learned by implementing (parts) of SQLite from scratch.
How does SQLite store data?
Difference between a "coroutine" and a "thread"
GPU code can now use Rust's threads. We share the implementation approach and what this unlocks for GPU programming.
Rust threads on the GPU
Support ZSF with a small recurring donation: https://www.every.org/zig-software-foundation-inchttps://github.com/sponsors/ziglang0:00 Intro2:20 package manag...
Apache Iggy benchmarking
Books in Progress is what we call a “public drafting tool”: Drafts will be made available for comment from the public, allowing for direct collaboration between author and reader.
The Soul of Maintaining a New Machine
Increase the scale of your PostgreSQL and MySQL deployment by up to 100x with Readyset - all without modifying your application code or database. Start using Readyset today for free!
How much Rust's bound checking actually costs
A collection of articles and videos to understand Golang internals. - emluque/golang-internals-resources
Writing a SQL Database, takes two: Zig and Rocks DB
Andy Grove blogs
Twitch is the world's leading video platform and community for gamers.
Instead of just throwing these objects after each use, which would only give the garbage collector more work, we stash them in a pool (sync.Pool). The next time we need something similar, we just grab it from the pool instead of making a new one from scratch.
RFC 9849: TLS Encrypted Client Hello
In this video I will demonstrate how you can setup kernel debugging with QEMU and explore how the kernel boots.Checkout my channel welcome link for informati...
Nick Licthman
How query engines work
In this talk, Jon Gjengset explores the true cost of concurrency coordination – from Amdahl's law down to CPU cache lines – and what we can do about it. Mode...
Cost of Concurrency Coordination with Jon Gjengset
Product quantization (PQ) is a popular method for dramatically compressing high-dimensional vectors to use 97% less memory, and for making nearest-neighbor search speeds 5.5x faster in our tests.
A basic intro to
On distributed systems broadly defined and other curiosities. The opinions on this site are my own.
GitHub Gist: instantly share code, notes, and snippets.
Understand Atomicity (the "A" in ACID) in databases, file systems, and hardware. Learn how it ensures data integrity and prevents partial updates.
Decoding Atomicity - The A in ACID
Efficiently computes derivatives of NumPy code. Contribute to HIPS/autograd development by creating an account on GitHub.
Autograd - automatically differentiate native Python and Numpy code
Karl Seguin's blog
2024 | 2023 | 2022 | 2021 2020 | 2019 | 2018... Read More +
Carnegie Mellon Database Group publications
Introduction Go is getting more and more popular nowadays, one of the reasons for that is great support of concurrency. Channels and gor...
Let's open a hex editor and see what this thing is made of
SQLite Internals: Pages & B-trees
A practical comparison of build and test speed between C++ and Rust.
Is coding in rust as bad as CPP
Io Uring
This multi-part series introduces Asynchronous Programming and the Twisted networking framework. In Which We Begin at the Beginning Slow Poetry and the Apocalypse Our Eye-beams Begin to Twist Twist…
Dive into database isolation! Understand its importance in maintaining data integrity during concurrent transactions with real-world examples.
Decoding Isolation - The I in ACID
Justin Jaffray blog posts
Mark Callaghan, Small Datum blogs