fig (aka:[phil])

@bad-example

art and transistors, they/them plant-mom, community infra for the atmosphere @microcosm.blue making/running constellation.microcosm.blue, slingshot, UFOs, spacedust, lightrail, relays, plc mirrors, hubble, service SLA & atproto consulting available

608 Following3,437 Followers5,036 Posts

Bluesky

Is this your profile?Claim it and bad-example.dirty.social is yours.Claim profile

fig (aka:[phil])@bad-example· 21d

waow 50% less memory for a mimalloc -> jemalloc swap (yes that is under-3G-rss while backfilling)

hubble memory rss

flat at 6G until just after 10:55, then a small gap, then it picks up at around 3G with an initial small rise, drop, and then probably stabilization at just under 3G

fig (aka:[phil])@bad-example· 22d

concurrency? yeah we got that (same machine)

actors working (stacked)

a very spiky busy chart, y-axis 0–over 1000 over 3 hours. a "ScheduledResync" (purple) task dominates, but a lot of "Firehose#commit" can be seen underneath. "Bootstrap#wake" makes up most of the rest, with some smaller things also happening but which are harder to make out.

fig (aka:[phil])@bad-example· 22d

hubble-sync doing full-network (full-content!) backfill on a lil 4 core 8GiB machine, using less than 4GiB 😌

hubble memory rss (line chart)

a slightly noisy line that initially has some peaks just over 4GiB, but tapers slightly to staying between 3 and less than 4GiB

over about 3 hours

fig (aka:[phil])@bad-example· 29d

😌

table-rendered json ('desynchronized_by_reason' is a subsection):

synchronized 75
desynchronized 24320
deactivated 0
gone 0
desynchronized_by_reason
app_requested 0
firehose_account_desync 0
firehose_account_throttled 0
firehose_fail 0
firehose_sync 0
first_seen 24320
future_rev 0
sync11_lax 0
throttled 0
unresolvable_identity 0

fig (aka:[phil])@bad-example· 29d

nothin just watching the backfill go by (single pds upstream for chiller vibes)

fig (aka:[phil])@bad-example· Jul 3

i am assembling a team

$use jacquard_axum::{ExtractXrpc, XrpcErrorResponse}; use jacquard_common::xrpc::XrpcError; use hubble_sync::{AccountStatus, CommitObject, Did}; use rocksdb::{IteratorMode, ReadOptions}; use star_lite::{CarBuilder, ConvertOptions};$

fig (aka:[phil])@bad-example· Jun 24

OOOHHHHhoho today is my DAY!!! i have a mild obsession with this central word, because it as an ABSURD number of different meanings

fig (aka:[phil])@bad-example· Jun 16

the most anti-climactic code i've ever written

// 6. check the commit's prevData against ours
if commit.prev != self.prev {
return Sync11Outcome::Desync;
}

fig (aka:[phil])@bad-example· Jun 14

post-game regret..?

account deletion rate and jetstream event rate: there's a big spike last night in the event rate from the ny nicks win, which was about halfway decayed by midnight. account deletion rate then spikes before 1am for a bit

fig (aka:[phil])@bad-example· Jun 11

ooohhhh yeah baby that's microcosm powering your mu.social verifications hehehe

screen crop of mu.social on a eurosky.social post with verification badge shown: the network tab is open, with details of a request:

Host: constellation.microcosm.blue
Filename: /xrpc/blue.microcosm.links.getBacklinks
--
source: app.bsky.graph.verification:subject
subject: did:plc:ooensn4mr5mhznzypvxelfa3

fig (aka:[phil])@bad-example· Jun 10

in my apply-for-things era 💅

not accepted
not accepted
not accepted
deadline missed
submitted

fig (aka:[phil])@bad-example· Jun 7

one neat thing about atproto’s merkle search trees with tid rkeys is that you get large, very stable subtrees. from some analysis i did on this last fall: more than 80% of layer-4 mst nodes tend to survive more than two months (active users) >90% for layer 3

sample 1: 20MB -> 21MB CAR snapshots

70-day survival rate

records (54,436): 99.97% (-14)
layer 0 (10,217): 99.82% (-18)
layer 1 ( 2,501): 99.32% (-17)
layer 2 ( 622): 97.75% (-14)
layer 3 ( 177): 94.92% (- 9)
layer 4 ( 39): 82.05% (- 7)
layer 5 ( 9): 66.67% (- 3)
layer 6 ( 2): 0.00%
layer 7 ( 1): 0.00%

ignored ( 888): 99.14% (- 4)

fig (aka:[phil])@bad-example· Jun 5

still pretty sure this SOULD and that MUST are not compatible bout to get active on that ietf wg mailing list muah ha ha

Producers SHOULD emit blocks in pre-order traversal of the included repository portion: header, commit object, root MST node, then a recursive depth-first interleaving of subtree nodes and the records they reference.

Preorder traversal enables streaming verification of repositories, allowing parsers to walk the MST structure and output key-to-record mappings while maintaining minimal MST state in memory. This approach supports efficient stream processing of large repositories without requiring complete buffering of the serialized data.

Parsers MUST tolerate other block orderings, duplicate occurrences of the same block, and additional unrelated blocks. Specifically:

fig (aka:[phil])@bad-example· May 31

[aliens meme] *actors*

box diagram: RepoActor (a tokio task)

external entry: registry.send - finds or wakes the repo actor

then all inside the actor:

intake rx: lil mpsc mailbox, goes to

intake loop: overflow policy, push or coalesce tasks, goes to

task queue, an internal VecDeque, to

process loop: actually runs tasks, does repo state mutations, with:

app callback, storage atomic write batch

fig (aka:[phil])@bad-example· May 21

last week i got exactly one brotli response from mushrooms, rest uncompressed, vs some indies for comparison (data from the same run)

milkcap.us-west.host.bsky.network (none) 448580
morel.us-east.host.bsky.network (none) 47256
mottlegill.us-west.host.bsky.network (none) 418860
mycena.us-west.host.bsky.network (none) 418703
oyster.us-east.host.bsky.network (none) 482727
oysterling.us-west.host.bsky.network (none) 417003
panthercap.us-east.host.bsky.network (none) 446256
panus.us-west.host.bsky.network (none) 424458
parasol.us-east.host.bsky.network (none) 445929
phellinus.us-west.host.bsky.network (none) 151380
pholiota.us-west.host.bsky.network (none) 515311
pioppino.us-west.host.bsky.network (none) 420639
poisonpie.us-west.host.bsky.network (none) 421514
polypore.us-west.host.bsky.network (none) 419107
porcini.us-east.host.bsky.network (none) 156186
psathyrella.us-west.host.bsky.network (none) 425332
puffball.us-east.host.bsky.network (none) 482488
reishi.us-east.host.bsky.network (none) 447059
rhizopogon.us-west.host.bsky.network (none) 312673
rooter.us-west.host.bsky.network (none) 147911
russula.us-west.host.bsky.network (none) 220797
scalycap.us-west.host.bsky.network (none) 416729
scarletina.us-east.host.bsky.network (none) 447930
shaggymane.us-west.host.bsky.network (none) 448767
shiitake.us-east.host.bsky.network (none) 81535
shimeji.us-east.host.bsky.network (none) 483372
splitgill.us-east.host.bsky.network (none) 332322
stinkhorn.us-west.host.bsky.network (none) 419532
stropharia.us-west.host.bsky.network (none) 611182
stropharia.us-west.host.bsky.network br 1
suillus.us-west.host.bsky.network (none) 418673
truffle.us-east.host.bsky.network (none) 446280
velvetfoot.us-east.host.bsky.network (none) 446359
verpa.us-west.host.bsky.network (none) 194347
waxcap.us-west.host.bsky.network …

"SELECT r.host,
COALESCE(m.content_encoding, '(none)') AS encoding,
COUNT(*) AS n
FROM repo_measurements m
JOIN repos r ON r.did = m.did
WHERE r.host IN ('selfhosted.social', 'northsky.social', 'skeetcentral.com')
GROUP BY r.host, encoding
ORDER BY r.host, n DESC;"
host encoding n
----------------- -------- ---
northsky.social br 291
northsky.social (none) 27
selfhosted.social br 557
selfhosted.social (none) 9
skeetcentral.com br 124

fig (aka:[phil])@bad-example· May 17

a solitary tree on a small front lawn, covered in blooming pink flowers, obscures a small blue house. two bright green trees are behind, and a close dark green tree borders the right edge of the frame. in the far distance is the side of a mountain with more trees, but the top quarter of the mountain background is obscured by solid cloud

fig (aka:[phil])@bad-example· May 12

the stats backfill for hubble hardware planning is what really enabled this: the entire non-bluesky atmosphere of repos is currently just 50GiB as raw CAR files, and under 10GiB compressed. hard to justify *not* taking some snapshots.

non_bsky_repos: 77,663
raw_car: 50.1 GiB
starlite_zstd3: 9.7 GiB
blobs: 4.5 M
blobs_size: 2.7 TiB

fig (aka:[phil])@bad-example· May 11

hey @snarfed.org, does this 429 from bridgy mean something other than "retry later", for backfill purposes?

Too Many Requests

temporarily disabled 12 hrs after repo creation

fig (aka:[phil])@bad-example· May 11

disappointingly, i can't pin this down to anything on the host. it had some further 14m45s-periodic throttles later yesterday. eventually the connection would get killed somewhere upstream and jetstream would restart. nothing on the host itself indicates a source of throttling though!

consumer_events_processed rate

log scale line chart, "commit" events are a yellow series that never actually goes to zero during "dead" times -- just drops to ~6 events per second. then eventually jumps up high to catch up, and settles at live-firehose steady-state, and then ... for some reason... drops to 6evt/s for a bit again.

fig (aka:[phil])@bad-example· May 10

hey thanks for the report! haven't looked into root cause yet, but my bsky-relay-listening jetstream (which slingshot currently using) did have a disruption-then-catchup clearly visible from the logs not sure if that was upstream from the bsky relay or from this jetstream instance itself

jetstream-bsky-network going to zero emit rate between 7am and 9:45am, then emitting at a high rate to catch up by 10am.

fig (aka:[phil])@bad-example· May 6

...finally got a fix in for this counter too, as a bonus

fig (aka:[phil])@bad-example· May 6

looking good...

screenshot: Links to did:plc:hdhoaan3xa3jiuq4fg4mefid

7 links from sh.tangled.graph.vouch:..

Links, most recent first:

[one sample shown]

fig (aka:[phil])@bad-example· May 4

backlogging moment

"operation": "create", "collection": "sh.tangled.graph.vouch"...
"reason": "really lovely person with nice vat pics "...

"operation": "update", "collection": "sh.tangled.graph.vouch"...
"reason": "really lovely person with nice cat pics "...

fig (aka:[phil])@bad-example· May 2

with a few more hours of data, zstd 9 compression, bigger output bundles (marginal improvement) (neat that you currently only need a 200K-entry cache to get ~90% hit rate on DIDs)

same as the last chart, just with the longer flatter trends extended over.

the compression performance gets a tiny bit better (under 30% estimated sztd bundled output w implicit cid), did hit rate breaks 90%

fig (aka:[phil])@bad-example· May 1

confronted with constellation being an actual part of the ecosystem today, whew - random posts in my for-you talking about pulling something off "without microcosm"(!) - worries about backlink detection for tangled vouches(!) - glancing at this chart again anyway: github.com/sponsors/uni...

estimated client apps (stacked)

constellation growing from about 80 to around 130/day
slingshot (starting around march) stacked on top, taking it from ~150 to ~200 daily active apps.

fig (aka:[phil])@bad-example· May 1

Couldn't resist looking back at firehose optimization after getting STAR-lite mostly working. Looks like 3x firehose bandwidth reduction is achievable with implicit MST + previous-commit cache of CIDs per repo.

two scatter charts:
first has two paired series: "raw: cid cache" + "raw: implicit cid" (second is smaller by a constant ~6%); and "zstd cid cache (+bundled) + zstd: estimated implicit", again the second improves by a roughly constant 6%.

both follow a sort of exponential decay toward a limit: first pair bottoms out around 60%; second around 33%.

the second chart has "did-cid cache hit ratio" and "overall cid hit ratio". both are kind of opposite of the last chart: rapid initial rise, slowing to some apparent limit.

did-cid cache: corners around 65% but keeps growing to around 80%, hasn't quite levelled off. overal cid hit ratio: levels at about 30%. might grow a bit more but not much.

fig (aka:[phil])@bad-example· Apr 30

make that > compresses just over 2x better than CAR and > over 6x smaller than uncompressed CAR if you go long with zstd-22

three bar charts, each with multiple series:

Compressed size vs uncompressed CAR (series: CAR, blue; STAR-lite, red)
Y-axis: 0–100%.
X groups: uncompressed, zstd --fast 1, zstd 3, zstd 9, zstd 9 +long, zstd 22
CAR starts at 100% (by definition) and lands just under 33%. there isn't a big change from zstd3 through zstd 22.
STAR-lite starts at a bit over 75% and lands around 16%. there's a bit more improvement at the higher zstd levels.
The final bars of each series are labelled: 3.09x (CAR), 6.29x (STAR-lite)

Bucketed STAR-lite / CAR by compressor
Same percentage Y-axis, series are each compression level including uncompressed.
X groups are: "< 100KiB", "100 KiB–1MiB", "1–10MiB", "10–100MiB", ">= 100MiB", "weighted average"
Uncompressed series: about 70% for <100KiB, levels off around 75% over the other buckets
zstd --fast 1: about half-way between uncompressed and zstd-3 for all buckets
zstd 3: just over 50% uncompressed, rising to perhaps 55% by 10–100MiB. bit over 50% weighted average.
higher compressions start very flat (little benefit over zstd 3) at <100KiB, and show more spread as the CAR size increases. At >=100MiB, zstd22 takes a dramatic drop to about 34%.
The weighted average shows a pretty clean spread: zstd3 is much better than zstd --fast 1, zstd9 is a little better, zstd 9 +long is similarly a little better, and zstd 22 is better again by a bit more.

Bucketed count of repos & weight (total size on disk)
X-groups: same buckets without weighted average
Left Y-axis and "count of CARs" series: power-curve rapid dropoff: about 350K repos < 100KiB, fewer than 100K 100K–1M, much less than 50K 1–10M, barely visible bar 10–100M; invisible at >= 100M
Right Y-axis and "total bucket size" series (the weight of the weighted average): <100KiB has the smallest total, less than 10GiB; then the last bucket, >=100MiB has slightly more. Middle buckets in order: 100K–1M a bit over 25GiB; 1–10M around 90GiB; 10–100M almost 100GiB

fig (aka:[phil])@bad-example· Apr 30

and i guess the buried lede: almost 2x smaller than CARs for the same compression settings; almost 6x smaller than raw CARs with zstd.

CAR vs STAR-lite for various compression configs: [bar chart]

raw: car 1x (by definition… it’s the comparison point). star is about 0.75x the size

gzip is around 1/3 for car and under 0.25 for star

zstd 3 and 9 are similar, a bit better for both (star well under the 0.25x line, hard to judge further kind of)

fig (aka:[phil])@bad-example· Apr 29

about to do a very boring demo! (which might be very interesting to a certain kind of person, several of whom i think will be there!)

overall (N=4866, raw CAR=2.18 GiB, raw STAR=1.62 GiB):

setting star vs raw CAR star vs same coder
mean med wt mean med wt

────────────
raw 0.668 0.678 0.746 — — —
gzip 0.292 0.232 0.215 0.568 0.556 0.552
zstd --fast 1 0.333 0.295 0.286 0.632 0.635 0.671
zstd 3 0.281 0.224 0.195 0.566 0.551 0.553
zstd 9 0.276 0.218 0.183 0.562 0.544 0.542

< 10 KiB (N=2168, raw CAR=6.31 MiB, raw STAR=4.04 MiB):

setting star vs raw CAR star vs same coder
mean med wt mean med wt
────────────
raw 0.614 0.611 0.640 — — —
gzip 0.395 0.370 0.301 0.624 0.640 0.585
zstd --fast 1 0.413 0.402 0.342 0.653 0.663 0.628
zstd 3 0.383 0.363 0.295 0.621 0.630 0.578
zstd 9 0.381 0.361 0.292 0.622 0.634 0.578

10 KiB – 1 MiB (N=2346, raw CAR=379.92 MiB, raw STAR=276.84 MiB):

setting star vs raw CAR star vs same coder
mean med wt mean med wt
───────────
raw 0.706 0.710 0.729 — — —
gzip 0.209 0.210 0.208 0.517 0.520 0.526
zstd --fast 1 0.266 0.270 0.270 0.607 0.615 0.629
zstd 3 0.199 0.198 0.193 0.519 0.521 0.529
zstd 9 0.193 0.191 0.184 0.511 0.513 0.519

1 MiB – 100 MiB (N=352, raw CAR=1.80 GiB, raw STAR=1.35 GiB):

raw 0.749 0.746 0.750 — — —
gzip 0.216 0.216 0.216 0.556 0.557 0.557
zstd --fast 1 0.288 0.290 0.289 0.676 0.675 0.680
zstd 3 0.192 0.194 0.195 0.547 0.543 0.558
zstd 9 0.180 0.182 0.183 0.534 0.532 0.547

fig (aka:[phil])@bad-example· Apr 17

same