macOS Sequoia Disk Space...Vanishing!

My everyday desktop is a 2018 Mac Mini. Last summer, I started noticing that some apps would crash overnight, mostly Ivory (my Mastodon client).

I assumed it was a memory thing. My disk space wasn’t great, but I had like 5 gigabytes or so free on the 256 gig drive, and wasn’t seeing any “your disk is full” errors. I wasn’t getting crazy “Your system is out of memory!” errors, either, but memory pressure seemed like a good explanation. I tried a few lazy tricks to get some data, try to collect logs, etc., but got nowhere.

Eventually, the problem seemed to move on to other apps. In September, I installed IoTuring, which let me send system stats to Home Assistant for graphing, but this didn’t tell me much, either. I even upgraded the RAM from 32 GB to 64, to make absolutely certain that memory pressure wasn’t the problem. But the crashes continued.

Not Haunted

It’s not just this machine that’s misbehaving. Apps were literally just disappearing on our Macbook Air. I’d be in Safari, swipe to a different screen to look at Preview, and when I swiped back, Safari would be closed. No warnings…the app was just gone. Finally, I logged out other users (my kids) and the system got a lot more usable. I eventually turned off my Messages cloud sync on that device only and it’s been fine ever since. (I have a lot of data in Messages, and needed to free up space to upgrade to Sequoia).

I know iOS will kill apps, without warning, due to memory or disk space issues, but hadn’t been aware that macOS did, too. The overnight crashes were annoying, but things were starting to interfere with day-to-day usage, like not seeing recent pictures in the Photos app. And it was getting worse - occasionally, Mail would crash with a “Can’t do anything! Your disk is full!” error (I’m paraphrasing here). Or Time Machine backups would fail. Or I’d see this popup a few times in a day:

The problem is definitely disk space. Occasionally, the disk would get legitimately full, but it usually hovered around 5 gig free. Which I’d think should be fine, but…maybe not?

Alright, fine. I’ve been called a hoarder before, and I know I have data that doesn’t need to be here (like 15 years of Messages and attached photos). So recently, I cleared out some cruft, got the free space up to about 12 gig, and called it a day.

Not so fast

Within a week, the problems had returned. And I cleared more space. I also noticed a lot of discrepancies. Being an old fart, I usually turn to command-line tools df and du for disk and directory space usage. But there’s also nice graphical tools like OmniDiskSweeper (which I like ‘cause it reminds me of DarkForest from NeXTSTEP). And there’s a storage pane in System Settings that tries to categorize just what’s in use.

Unfortunately, these tools all showed me different results. In the end, I was never quite completely sure where all this disk space was going. Is it in my home folder? application caches? something at the OS level, or files that are deleted but never properly purged? …or… something else?

Finally, last Friday, I’d had enough. I had recently cleared some space, and a couple days later, was down to under a gig free. So I spent the afternoon exploring.

Spotlight!

Of course, it’s spotlight. I think. Maybe? I found some data in ~/Library/Metadata that didn’t look right. A few gig in CoreSpotlight (okay, that’s probably reasonable), and THIRTY SEVEN GIGABYTES in a folder called SpotlightKnowledgeEvents. Wow.

So I deleted that folder. I also Google Earth cache data (and adjusted the app’s settings), and moved some more files to external storage, but that didn’t save nearly as much. Then, I rebooted, and boom!, I’ve got 43 gig free. Whew! That feels much better. Over the next couple hours, that space got reduced by about 9 or 10 gig, I figured mostly to rebuilding the actual Spotlight index, but I wasn’t too worried.

Until I returned to the computer the next afternoon – and another 19 gig had vanished. What the hell? Am I really losing a gigabyte an hour? Is the file system just messed up, and I need to wipe & rebuild? I’d really like to avoid doing that – I’d much rather figure out what’s actually happening.

I posted a plea for help in a couple places online, and walked away.

It’s Time Machine, duh.

Almost immediately, I realized – “wait, it’s probably Time Machine snapshots.” I had even cited that as the likely source of the spikiness on the graph. But even if snapshots are the problem … why am I generating 20 gig of “noise” when the computer is just sitting, locked and idle? What is actually being backed up?

I returned to the problem on Monday, asking myself: “How big are the snapshots, anyway?”

Turns out that’s not easy to tell.

The command line utility diskutil has a command (diskutil apfs listsnapshots disk4s1) that, predictably, lists all snapshots for a given disk. Unfortunately, it doesn’t include sizes. The DiskUtility application, however, does:

It looks like that “Private Size” column is what I’m looking for. Though the snapshot may appear to be a complete copy of the filesystem, it only takes up as much space as is needed for what’s different from the current filesystem itself.

I proved this to myself by checking free space, deleting a snapshot, then checking space again. It was … close, but not quite exact. The space actually freed up was more than the Private Size would’ve indicated. This was especially true when I removed all the snapshots. I repeated the text the next day. Here’s a rough table of what I learned:

Snapshot	Private Size	Space Recovered
2/16 10:46	108.5 MB	113 MB
2/16 19:42	159.1 MB	152 MB
(all - Monday)	1.9 GB	6.17 GB
(all - Tuesday)	1.73 GB	8 GB

Individually, the sizes seem to make sense. Taken together, it’s a lot more - which may just have to do with how snapshots accumulate. I’m not sure.

How about the question, “What’s getting saved in these snapshots?” For that, I tried a couple approaches:

• Mount two different snapshots on the filesystem and use an rsync command to show the diffs
• Use tmutil-compare to compare two different backups

The second approach worked best. Taking the output and sorting by the first column (size), I get something that looks like this (long paths omitted for barely-improved readability):

$ tmutil compare -@st /Volumes/TMRohan/Backups.backupdb/rohan/2025-02-17-011314 /Volumes/TMRohan/Backups.backupdb/rohan/2025-02-17-021400 | sort -h -k 2 -r | grep size | head
! 384.6M  (size, mtime)         [backup path] - Data/private/var/folders/_q/27430_pn4b98ywtfwchsdtsc0000gn/0/com.apple.ScreenTimeAgent/Store/RMAdminStore-Local.sqlite
! 322.9M  (size, mtime)         [backup path] - Data/Library/Backblaze.bzpkg/bzdata/bzfilelists/v003df00d10839a8cede8dd70114______filelist.dat.future 
! 9.6M    (size, mtime)         [backup path] - Data/Library/Backblaze.bzpkg/bzdata/bzlogs/bztransmit/bztransmit17.log

    [...and more...]

You also get a handy summary at the end:

Added:         18.0M
Removed:       46.3M
Changed:       891.8M

The big files were for ScreenTime, Backblaze, and some obvious system data. I did this again, comparing midnight to 8 am, and had similar results. So I’m not really concerned that some process is generating lots of noise that’s bloating the hourly snapshots.

So anyway, I deleted the snapshots, restoring 6 gig. But there was still another 13 gig that I’d lost (remember, I went from 43 to 33 to 14). Where is that? (Spoiler: I think I figured it out. But first…should I do anything about these snapshots?)

Thinning Snapshots

Time Machine makes a new backup every hour. First it creates a local snapshot of the filesystem, then copies that to an external drive. It keeps snapshots around for a day (so you have at most 24 snapshots), and deletes them as needed to keep disk space free. I could trust the OS to handle this for me, if I didn’t mind seeing my space hover…around 10-15 gig free. But if the data’s already on the external disk, I just want it gone – I’d rather have the space available. I probably don’t need the space available, but I’m just being… paranoid, I guess.

This was never a consideration before, because of that 37 GB folder I nuked the other day – there was never enough space to keep more than 1 or 2 snapshots around, at most.

There’s no way to not keep snapshots, or to set a limit of say, 10 GB of snapshots, so I added this to my crontab:

# try to remove up to 5 gig of time machine backup snapshots, urgently (4 = highest urgency)
# do this twice - in morning, and early afternoon. Hopefully if there's a problem with the
# external drive, I'll have noticed a pop-up and can fix it before the 2nd batch of cleansing occurs
#
15 06 * * * /usr/bin/tmutil thinlocalsnapshots . 5000000000 4  
15 14 * * * /usr/bin/tmutil thinlocalsnapshots . 5000000000 4  

Ideally, I’d call a script, and do some housekeeping and quality checking first. Like…is the external drive even mounted? Have these snapshots all been copied to it? And so on. Maybe I’ll do that eventually. For now, I think I’m confident enough that I’ll notice a “The time machine backup failed!” warning when I sit down in front of the computer.

The Culprit Is…

(Imagine that said as Hercule Poirot. I could never do Poirot. I might pull off a passable Benoit Blanc, but it’d probably sound like Foghorn Leghorn. I digress.)

Let’s return to that spotlight data in ~/Library/Metadata. There are two folders I’ll focus on: CoreSpotlight and SpotlightKnowledgeEvents. On Friday, those folders totaled about 40 gigabytes. After rebooting, CoreSpotlight quickly returned to a similar size. The SpotlightKnowledgeEvents folder grew to about 4.6 by the end of the day, and to 9.1 by Saturday afternoon. In a moment, I’ll explain exactly why I think that’s the problem. But first…

Additional-but-Helpful Digression: Does this all add up?

Let’s see if I can sum up the last few days (going back a little more, just to illustrate how widely things were fluctuating every day).

(commences deep dive through terminal history across a half dozen different windows, notes, frustrated text messages to my wife wherein I complain about this damned problem, and other sources of information)

When	Core Spotlight	Spotlight Knowledge Events	Snapshots	Space Before	Actions Taken	Space After
Wednesday noon	?	?	?	2.4 GB	Rebooted (clearing app caches, etc.?)	8.8 GB
Friday afternoon	3 GB	37 GB	?	< 1 GB	Delete Google Earth cache, personal Spotlight data, reboot	43 GB
Saturday afternoon	1.5 GB	4.6 GB	11 snapshots, ~7 gig	14 GB	Disable all spotlight indexing, clear snapshots.	41 GB
Monday morning	1.7 GB	9.1 GB	23 snapshots, ~6.7 GB	27 GB	Deleted snapshots, reboot	34 GB
Monday midday	1.7 GB	9.1 GB	0	34 GB	Deleted Knowledge Events, tweaked settings (see below)	43 GB
Tuesday morning	1.5 GB	110 MB	20 snapshots, ~8 GB	32 GB	Deleted snapshots	40 GB
Tuesday evening	1.5 GB	121 MB	7 snapshots, ~3 GB	36 GB	Deleted snapshots	39 GB
Wednesday morning	1.5 GB	124 MB	3 snapshots, ~1 GB	21 GB	Deleted snapshots	22 GB
Wednesday mid-morn	1.5 GB	128 MB	0	22 GB	Closed everything but Safari & Sublime	38 GB

I’m not entirely sure what else I did on Saturday - there’s about 20 gig unaccounted for. Unless it was all spotlight indexing across the entire system. Somehow, I went from 43 gig free, down to 14 (with the computer locked and idle), then after deleting a few things, back to 41. (I wasn’t taking detailed notes at this point, just…fumbling around in the dark still). (“Remember kids, the only difference between screwing around, and science, is writing it down.” - Alex Jason, Mythbusters ballistics expert, via Adam Savage)

But aside from that bit of confusion….I seem to be reliably getting the system back to the 40 gig free range.

That’s a pretty impressive fix. What changed?

Two things:

• Thinning snapshots (which should now be happening automatically twice a day)
• Clearing out SpotlightKnowledgeEvents

Also, it’s clear that some of my apps are also bad about caching. Since I have Slack, Discord, and Wire all running continuously (and they’re all Electron apps), this probably shouldn’t surprise me much. But 16 gig restored this morning, just by closing those apps? That’s insane. I’d force a quit and relaunch of all those every day, if I could find a way to ensure they go back to their proper monitor / space afterwards.

But electron cache issues are easy to fix - quit & restart. The real problem is when I do all that, and the disk is still full….which brings me back to the Revelation in the Parlor.

(the culprit is….) SPOTLIGHT KNOWLEDGE EVENTS!

(lightning flashes, thunder rumbles, lights momentarily dim)

At this point, I’m feeling confident that ~/Library/Metadata/SpotlightKnowledgeEvents is the real source of my problems. Evidence:

1. It was massive (37 GB!) when I started digging into all this on Friday
2. I found several other people complain about it online (yes, that’s anecdata, but it’s something)
3. The complaints all seem to focus on Sequoia. Though I’ve had issues back to the summer, I resolved most of those by deleting junk here and there, moving things to external drives, etc. This didn’t truly become untenable until sometime in the fall. I upgraded to 15.0 on September 18. A quick summary (from other notes) of efforts to keep the drive clear:
   • 8/7 - updated OS to 14.6, 20 gig free afterwards
   • 9/5 - under 4 gig free, TM failed
   • 9/18 - upgraded to 15.0
   • 10/21 - Receiving system disk full errors. Emptied trash, closed apps - 2.6 gig free. Rebooted - 12 gig free.
   • 10/28 - moved photos library to external drive, freeing up about 12 gig
   • 11/20 - moved about 16 gig of bulky data to external drive (that’s 28 GB moved to external drives in a month)
4. For all the complaints I’ve found online, when they mention a specific Mac model, it’s using an Intel processor, just like my Mini. This folder doesn’t even exist on the family’s iMac or Macbook Air (both of which use M1 chips).
5. After deleting it on Friday, it grew to over 9 GB in a day.
6. Since tweaking some Spotlight settings and rebooting, the folder has barely changed

Okay, but what’s the folder even for? At first glance, a lot of it looks like spotlight index data (the top folder is “index.V2”). Eventually, you find your way to a folder full of .journal files, and that’s where it gets interesting. Before I cleared it out, it had 71 journal files. Most were small (under a hundred MB) but several were very large, including one that, by itself, was almost 2 gig.

Of the 71 files, 56 contained email. Not just subjects or metadata, but what looked like full text (these are proprietary binary files with lots of other structure to them, so I’m just poking around using xxd and strings). Others were Messages messages, or Calendar event data, or…other stuff.

At some point on Monday, I went to Spotlight in System Settings, and disabled “Mail & Messages” from the Search Results list. (interestingly, this is worded as “Only selected categories will appear in Spotlight search results” - does this mean they’re indexed but not displayed, or not indexed at all? Precision in language matters, Apple.)

I was also going to turn off “Learn from this application” for Mail, under Siri’s settings, but didn’t want to change two things at once. At this point, it may not be necessary to even try.

I then deleted the SpotlightKnowledgeEvents folder, and rebooted. Within an hour, the folder was back to 110 MB. A day and a half later, it was 121 MB. There are still journal files there, but they seem to be getting updated much less frequently .

Note there’s also a SpotlightKnowledgeEvents folder under CoreSpotlight, but the size and number of files in that folder seem much more reasonable.

So what’s the deal?

I’m betting it’s a bug, likely on Intel systems only. (this is one reason I haven’t applied the 15.3.1 update yet - just in case that fixes it before I can gather more data).

What Next?

Here, at the end of a typical David-Wall-o-Text, is the simple summary:

• I’ve got way too much crap on my system, and always have. This was causing things to crash last spring and summer. I’d clean out cruft regularly, but never did a deep dive, and was always on the edge of problems.
• When the drive gets close to full (even if there’s literally multiple gigabytes left), apps will just randomly close with no warning
• After upgrading to Sequoia, the problem got worse. A bug (speculation) caused SpotlightKnowledgeEvents to rapidly fill up the drive, with 37 gig stored.
• I turned off Spotlight indexing for Mail & Messages, and the file seems to be stable at around 100 BM.
• Adding the thin snapshots to cron should also ensure I’ve got plenty of free space

What do I do next?

BREAKING NEWS - SMOKING GUN?!!

As I was doing a final review of this post, I thought to myself…“It’s too bad I didn’t get more detail on that SpotlightKnowledgeEvents folder before I deleted it.” Then I facepalmed – it’s on the Time Machine backup, stupid. sigh.

So, some interesting data. As of Thursday, 2/13, the folder SpotlightKnowledgeEvents/index.V2/journals/11/cs_default contained:

• 1136 .journal files
• The journals totaled 30 gigabytes
• The average size was 28 MB, and the largest was 1.1 gig
• The most recent file was created on February 13
• The oldest journal file was dated October 28, 14:32

That’s an interesting date. If you go to the System Information app (as I just now learned), you can get a list of all your software installation dates, for apps and operating systems. What happened on October 28, at 14:31? I upgraded to macOS 15.1:

But wait, there’s also a ....journals/10/cs_default folder. The oldest file there was dated: September 18, 14:34. This folder was only 2.7 GB, but then again, the files there only spanned a month (9/18 to 10/28), while the other folder covers almost 4 months. Looking again at my installation history, I’d upgraded to 15.0 just two minutes beforehand – the first Sequoia update. (Note that there’s also cs_priority in both 10 and 11, but those are significantly smaller. Just for completeness’ sake:)

% du -shc */*
2.7G    10/cs_default
159M    10/cs_priority
 30G    11/cs_default
298M    11/cs_priority

So this is definitely a problem (bug or not) introduced with Sequoia 15.0, possibly made worse with 15.1, and continuing until the present time.

So really, what should I do next?

First, I turned off spotlight indexing on all drives (including the internal drive). I need to slowly re-enable those, one drive at a time, and see how that affects things. Because, though I don’t use it a lot, I do use Spotlight to search my files.

And the next time I’ve lost a bunch of space, and have already cleared the Spotlight & snapshot troublemakers… I should take a snapshot, close all my apps, and immediately take a new snapshot. Then I can compare the two, to try and narrow down the offending files & applications. This morning’s restoration of 16 gig just by closing apps was kind of troubling. It’s possible I haven’t really figured it out after all.

I should probably look into other backup options as well. I use Time Machine as an “Oh, crap!” immediate recovery mechanism, when I mess something up and realize it quickly. And BackBlaze serves as an off-site disaster recovery backup (in case a lightning strike takes out both the Mac and the TM drive). But neither is really good for long term archival backups. I’d love to have, say, 2 months of weekly deltas, 6 months of monthlies, and 2 years of quarterly changes. So I’d still be able to recover that old file I accidentally trashed 3 months ago.

And in the long run, I’m hoping to upgrade to a new M4 Pro Mac Mini later this year. I’d like to think that the base storage (512 GB) should be enough, but if I can’t figure this problem out…maybe I’ll spring for the overpriced 1TB option.