Disk Usage Woes: Day…whatever. How long has it been since I installed macOS Sequoia?

I’ve been struggling with the disk constantly filling up on my Mac Mini. See my last two posts for a deep and wonky dive into the situation, and lots of data collection to confirm my suspicions (or not).

At this point, I’m still continuing to collect data, but it’s more in the realm of “just making more graphs for reference” than actually figuring anything out. I think the fix is pretty clear at this point: I need to delete extraneous data from time to time. I just now have a better idea where to find the best low-hanging fruit for deletion.

Smoking Guns

So far, there’s nothing solid, other than one of the first things I found back a couple weeks ago: Huge amounts of data in ~/Library/Metadata/SpotlightKnowledgeEvents. It was at 37 gigabytes when I first started this process, and after clearing it, was back at 10 gig the next day. Since then, the most I’ve seen is about 3 gig. But I still haven’t found anything to reliably prevent data from accumulating there.

But this isn’t the only problem. I’ve also seen, for months, disk storage fluctuate significantly over the course of a day, and sometimes, even minute-by-minute. I don’t have cold data for some of that, but I know, back when I was regularly checking disk space, that I’d sometimes see it drop by 2-5 gig, then 5 minutes later, all that comes back. It’s possible I was experiencing a real-time, terminal-based view of the spikes I’ve seen on disk usage graphs (which I’ll talk about later in this post).

Anyway, the way I see it, there are three key at play (aside from being a digital hoarder, which is a totally different problem I’ve kinda given up on solving for the moment):

• Spotlight drive indexing - This is just plain old “enabling Spotlight searches on the main drive.” Again, I haven’t been tracking this closely, but in the last few days I’ve seen the size of the disk index range from 2-3 gigabytes to nearly 30 gig. And I don’t think there’s much that can be done here other than just turn it off.
• SpotlightKnowledgeEvents - The big offender and presumed bug that started all this. It still seems to be happening, even after upgrading to 15.3.1. But it seems rarer. Since it’s a bug, it’s probably not going to be predictable at all – and maybe before I even finish this post I’ll find 10 gig there. Again.
• Time Machine local snapshots - These are perfectly reasonable and expected. The system does a good job removing them when the system hits 95% full – I just wish I had more control over them. Like, it’d be great to say “Keep the last 10 snapshots” or “keep as many as you like, up to 5 gigabytes” or something.

More data! (still!)

I’m continuing to track data, but I’m not testing as many settings as I do. To start with, let’s look at the drive usage for the past few days. Here, Spotlight indexing is turned on for the local drive, and I’ve disabled my crontab-based snapshot thinning process (so they disappear only when the operating system feels the disk is getting full).

New Clutter

This first graph shows quite a bit of fluctuation. There’s a gradual, consistent increase in disk usage, punctuated by a very fast and high spikes (which then just as quickly drop, just as much or more). It’s almost like the system slowly collects a lot of temporary data, then quickly process all that data into another temporary structure, then throws away ALL of the data (or compiles it into a much smaller result). Also seen at the far end of the graph (with no spike) is another sudden, large jump, after about 12 hours of slow and steady (and spike free) growth.

What exactly are these spikes, and what’s the lone jump at the end? For a few days, I left an instance of fs_usage running in a terminal window, filtering for just WrData calls. After taking this graph snapshot, I searched in this log for any suspicious-looking activity that matched the times of the spikes. What I found (working backwards):

• For the spike on 3/1 at 22:39: Over 2500 file writes in the drive’s .Spotlight-V100 index (out of 7700+ writes that exact minute). There were also a lot of Backblaze activity at 22:37, and Time Machine activity at 22:38. Also, a very large (I didn’t count) burst of mds_stores activity (Spotlight indexing) at 22:38.
• Surrounding the spike at 19:38: Lots of Spotlight activity seen at 19:37, 19:38, and especially from 19:43-19:45. Finally, a burst of Backblaze activity at 19:40.
• Going farther back (as far as my terminal backscroll went), I saw that the vast majority of writes from 17:58-18:02 were to the Spotlight drive index folder.

So it definitely feels like these spikes are related to either Spotlight, Backblaze, or Time Machine. Based on the sheer volume of logged writes, I’m leaning towards Spotlight drive indexing. Looking at graphs from the prior week (in my last post) I didn’t see any such spikes when drive indexing was disabled, but did see them when it was. Since Time Machine and Backblaze were never disabled, this again points to Spotlight being the source of the spikes.

What about that jump late in the morning on March 2? In that case, the jump is about 1% of a disk with 233 gigabytes of usable space, or about 2.3 gig. In the SpotlightKnoledgeEvents folder, I see 2.4 gig of data spread across 55 journal files. A large chunk of those files (including one file which was itself over a gig) were written between 11:06 and 11:16. So I’m pretty sure that’s what this is – the Spotlight bug at the root of this extended rabbit hole exploration. (I should note that at least one of these journal files had bits of email from 2013 – so it’s not like this is even triggering exclusively on new mail).

The Sweet Release of Deletion

So those are increases. Refreshingly, my data also show drops in usage. This graph shows 1% or 2% drops, three different times, with remarkably steady and unchanging disk usage in between and afterwards.

My guess here is that this is showing the effects of automatic Time Machine snapshot thinning, due to age (it retains, at most, the last 24 snapshots). In particular, my notes show that total snapshot usage dropped from 24.3 gig (midday on March 2) to about 8.9 gig (morning of March 3).

By rolling my mouse over the graph in the Home Assistant interface, I can see drops in the chart at 17:55, 18:54, 19:51, 20:51, and 22:51. In my terminal backscroll, I see Time Machine snapshots that had been taken at nearly those same times (17:49, 18:50, 19:50, 20:50, 21:51, and 22:51). The exact time of the drops in the graph is hard to pin down but it’s certainly within a couple minutes of those new snapshots being formed. So these are almost certainly old snapshots being dropped.

Spotlight Overall

What I hadn’t been tracking carefully was how much space the overall spotlight index for the main system drive takes. But after re-enabling it on the 26th, I noticed it got really big pretty quickly:

After it dropped on the 28th, usage remained very steady at about 3 gigabytes (plus or minus about a 100 meg) until I disabled and cleared the index on the morning of March 4.

Seeing 3 gig of index data for a 256-gig drive seems reasonable. Seeing close to 30 (over 10%) seemed… not. So on 3/4, I deleted the index and disabled Spotlight for the system drive, deleted the SpotlightKnowledgeEvent index, and deleted all Time Machine snapshots. I also closed all my apps, emptied the trash, and then rebooted. After reboot, I had about 44 gig free (wow!).

I decided to let it sit at that state for about a couple of days (with no spotlight indexing enabled) just to see how snapshot growth without spotlight looked, and to verify that I wasn’t seeing any sudden jumps or spikes. Here’s what it looked like - a nice, steady growth, largely as snapshots accumulate, then a fairly flat (if jiggly) steady state as old snapshots are replaced with new ones every hour.

Finally, I cleared everything one last time, rebooted, and re-enabled Spotlight indexing on the system drive. This should, hopefully, be my last long-term data collection, then I can start building some hacks to keep the numbers reasonable.

Growth in this graph is faster than in the last (you have to check the vertical scale to be sure). In both cases I cleared space & rebooted around 9:30 am. Without spotlight (above), usage had risen 6% by 6:00 the next morning. With spotlight (below), it had risen by 15%, in a similar time period, to trigger snapshot thinning at 95% usage. Also returning are the thin spikes associated with Spotlight indexing. Finally, a short period of (spikeless) growth around 4:00 am on 3/7 was matched to a 3.3 gig increase in SpotlightKnowledgeEvents.

All in all, these two graphs (with and without Spotlight) pretty much confirms the behaviors I’ve been seeing. Without Spotlight, the drive fills to about 86%, and 24-hour snapshot thinning keeps it near that level. With Spotlight, we get regular spikes from Spotlight activity. Those would be bad if my drive were already at 95% with no snapshots to clear – as it had been before I started down this rabbit hole. With nothing ready to purge by the system, this is probably why I had apps crashing overnight.

Spotlight & Snapshot Usage

There are two further bits of data that I’ve been tracking. They’re worth sharing, but not dramatic enough to detail extensively.

As I said earlier, the drive Spotlight index (/System/Volumes/Data/.Spotlight-V100) grew significantly to nearly 30 gig. But this time around, it peaked at around 7 gig, and has been fluctuating around 4-6 gig since. That feels much more reasonable, I’d think.

And the overall Time Machine backup snapshot size has been fairly constant in both tests. With spotlight off, I typically had a full 24 snapshots, with a total size (as reported by Disk Utility) of around 11 gig. With spotlight on, the number of snapshots has varied. I recorded between 7 and 13 snapshots on 3/7, with total space used in the 16 gig range. I was out of the house all day the next day, and since I can only see the total space in the Disk Utility GUI, I couldn’t track usage. The morning of 3/9, I was down to 6 snapshots of about 11 gig total. Given the volatility of things like the Spotlight index, staying in the range of ~10 (plus or minus) snapshots, for between 10-15 gigs, seems reasonable.

So…at this point I’ve analyzed the hell out of the various eaters of my disk space (Time Machine, Spotlight, and Spotlight Knowledge Events), and finally understand their typical demands. What next?

Fixes

I feel like thinning snapshots at 95% is too high, since the spotlight spikes can push disk usage into the “kill running apps” territory. If I could set a lower high-water mark, that’d be awesome, and maybe there’s a utility out there that’ll do that for me. Or I could just delete all but the last N snapshots, without regard to size, and experiment over time with the right setting for N. It’d sure be nice if I could better understand how tmutil thinlocalsnapshots works.

As for Spotlight - maybe I could use some other service? I’m not sure what’s out there, and anything else might not be better anyway. I’ll probably stick with Spotlight and just hope extra crazy stuff is rare. (As I write this, the drive Spotlight index has averaged 4.8 gb over 20 measurements, which is a lot, but not bad. If it were staying above 10 or 15 gig, I’d probably be more anxious to switch).

Finally, there’s SpotlightKnowledgeEvents, the thing that started (gestures to all of this). That’s been solidly 3.3 gig (plus or minus maybe 100 meg) since it showed up again. I can certainly live with that. I’ll just keep an eye on that folder, and if it starts to get out of control again, I can just delete it.

Actual Solution

So any long-term solution is coming down to some cron-based script. I’m thinking that, every hour, I want to:

• Record what I can to Home Assistant - the size of SKE and .Spotlight folders, number of shapshots, total disk usage. For more graphs. For Science.
• If the disk is over some base threshold (85% may be unattainable for my system - so maybe 90%), then:
  • Clear SpotlightKnowledgeEvents if it’s too big (using maybe a 5-10 gig threshold)
  • If the disk is still over that 90% limit, then thin Time Machine snapshots as described above

Takeaways

Let’s wrap this all up, shall we?

The Original Problem: This all started because I kept noticing applications crashing overnight (without warning or post-crash notification). Occasionally, backups would fail because the drive was full. Or Mail or Fantastical would crash (those do give me warnings, at least). I’d start watching my disk usage closely, and it could fluctuate widely – as much as a few (or even several) gigabyte swings in the space of a half hour or less.

SpotlightKnowledgeEvents: This index, in ~/Library/Metadata (different from a similar folder in Metadata/CoreSpotlight), seems to be the result of a bug on Intel systems, introduced in macOS 15.0 (Sequoia). When I started this journey, almost a month ago, I had nearly 40 gigabytes of junk in there. And I can’t seem to make it go completely away. I don’t know if these are temporary files built during indexing, that were saved in the wrong place (and so never deleted), or if they’re actually actively used. They seem to include (among other data) lots of email and Messages content – in some cases, years-old data.

Time Machine Snapshots: These are not entirely predictable – sometimes they’re really big, sometimes fairly small and simple. I suspect they’re bigger when Spotlight indexing is enabled, and they also include the SpotlightKnowledgeEvent data, so if that bug is causing trouble, it’ll affect backup sizes. The operating system automatically deletes these after 24 snapshots, or if we hit 95% disk usage. I’d like to reduce that threshold to 90% or less, just to have more flexibility.

Spotlight drive indexing: This also seems fairly volatile, especially during initial indexing. I’ve seen it jump to as high as 30 gig, dropping to only 3 gig without any direct action by me. I think I need to just…leave it be. This is part of why I want to make sure there’s lots of open space on the drive, especially since the indexing process creates very large, half-hour long, spikes in disk usage. Like Time Machine, it’s something I want to have enabled, so I just have to live with (or work around) the space usage quirks.

App caches: I haven’t dug too deeply into this, but there’s still some benefit to occasionally killing apps (as much as that feels superfluous, especially on iOS). Many desktop applications retain large local caches, or are memory hungry. Not to harp on Electron, but regularly restarting Discord & Slack is probably a good idea. (Never launching them in the first place is probably an even better idea, but that’s not really an option).

Other cruft: I definitely need to be better at clearing out old downloads, temporary stuff I don’t need anymore, application installers, etc. And I absolutely need to find time (and some kind of long-term archive structure) to clear out iMessage history and especially attachments – that’ll free up another 20-30 gb. (This would also let me re-enable it on the family laptop, which has even less free space available since it’s shared among 5 people).

The Future

Hopefully some future software update fixes the problem altogether. Well, for SpotlightKnowledgeEvents. It’s not likely that the snapshot behavior for Time Machine will change significantly, nor Spotlight, so in those cases it’s still down to me to monitor & delete data as necessary.

I should also keep monitoring – it may be that something else is also a problem, though at this point I’m feeling fairly confident I’ve found the three biggest contributors.

Is there a pattern to to the various jumps, spikes, and snapshot growth? Probably not. The volatility in spotlight activity would be tied to how much new data my system stores – and even when I’m not doing anything, there’s always messages and emails coming in. The disk space used by snapshots sizes is directly tied to both that new data, and to changes in spotlight indexes (which are also affected by the data). And for the SpotlightKnowledgeEvents data – assuming that’s a bug (and the data should be stored elsewhere, and automatically purged), then I’d probably never find a pattern.

So for the short term, a simple script called in crontab (every hour or so) will almost certainly be sufficient. In the long term, I’ll be replacing this machine anyway (it’s a 2018 model), and I’ll definitely be going for at least a 512 GB. Which may not solve the problem, but will certainly kick the can down the road for a few years.