My main day-to-day machine is a 2018 Mac Mini. Over the last year, and especially the last 6 months, it’s been a struggle to keep adequate space free. About two weeks ago, I’d had enough, and went on a deep dive to figure out what was wrong. In my last post, I described the journey I took to identify the likely problem.

Simply identifying a “likely” culprit wasn’t enough, though. I needed to take a methodical approach to testing different settings and recording the results.

The problem centered around a presumed bug in macOS Sequoia for Intel processors, which left dozens of .journal files in the ~Library/Metadata/SpotlightKnowledgeEvents folder. Many of these files can be hundreds of megabytes, and even nearly a full gig. When I discovered the folder, it was over 37 gig total.

Writing Stuff Down

Assuming that this folder is the primary source of my space issues, I had to determine whether a simple setting change could mitigate the problem. To that end, I focused on two different Spotlight related settings:

• Whether the main system disk is indexed by Spotlight (configured in the Spotlight settings pane, by adding or removing the drive from the “Search Privacy” sub pane)
• Whether “Mail & Messages” results are included in search results (again, in the Spotlight settings pane). It’s unclear if this setting actually disables collection and indexing of data from Mail & Messages apps, or if it simply disables seeing the contents of that index.

My hope is that I only see the disk usage balloon out of control when both Disk and Mail/Message indexing are enabled. (spoilers: That was…almost verified…maybe?)

For a little over a week, then, I tried all four possibilities. First, I cleared as much space as possible, then I changed the relevant Spotlight settings. I rebooted, then just…used my computer as usual. Periodically, I took measurements and shoved them in an ugly text file.

*Note – that 0.05 GB figure…no longer held true about 12 hours after measurement. We’ll get to that later.

Unfortunately, those numbers are all from whenever I remembered to run a manual set of space-checking commands in a terminal window (and checking the cumulative snapshot usage in DiskUtility.app). So the durations are all over the map (sometimes it’s like 30 hours, sometimes closer to 40). However….

Graphs! We’ve got Graphs!

I’m also tracking the overall percentage of disk usage in Home Assistant, so I’ve built graphs for a 30 hour period, one for each of the rows in the above table. All graphs show a period from 9:00 am until 3:00 pm the following day, for at least some consistency.

We’ll start with the first couple of days after the last blog post, beginning on February 19. Here, I’ve turned off Spotlight indexing for all my drives (internal and external), but left Mail & Messages results enabled. It shows a continual 45° slope up and to the right, but with sudden vertical drops at 14:15 and 6:15 (due to automated snapshot thinning). Note the sudden jumps about 4 hours after each thinning, “catching up” to the previous trajectory.

Next, I turned indexing of the main system disk back on. Mail & Message results remain activated, as before. Again, we see that sudden jump after each thinning. The graphs also shows 1-2% spikes every hour or so (for only a few measurements - about 15-20 minutes). The usage jumps up, and almost immediately drops back down, then the slope continues.

Then, I turned everything off – no Spotlight disk indexing for the main drive, and “return results” for Mail & Messages is off (which I presume means it’s also not collecting results for Mail or Messages¹). Now we see a much slower, almost flat, growth slope, and without the sudden jump back to previous trajectory after thinning. And there’s none of the previous spikiness at all.

Finally, let’s try the last combination: Spotlight on, but Mail & Message indexing disabled. This time, much like the prior graph, we see a continual slope upwards to the right. The 6 am Snapshot thinning is there, after which the graph continues with the same slope (but lower), as before. A little after 10 am, there’s a small upwards jump which I couldn’t pin down, and then a nearly equal drop about 2 pm, which I attributed to a couple backup snapshots being automatically deleted. (at that point, usage was right at 95%, so the operating system initiated this snapshot clearing). (also, there’s a gap of about two hours when I’d forgotten to restart my data logger after rebooting)

Forget about graphs. Graphs LIE!

Just for fun, let’s look at the data for the whole week (minus the last test configuration - I’m assembling this post as I collect data, so you get a sense of some of the thrills of the process). This graph shows the 19th through the 25th. Note in particular the periods beginning the morning of February 19 and 24. You may also notice how usage jumped, to noisily hover in the 90-95% range, from the 22nd through the 24th, when Spotlight and Mail indexing were both active. Usage also quickly jumped back to that range after a brutal thinning of backup snapshots.

In this view, the mornings of the 19th and the 24th look very similar. But in the other graphs, the 24th is very slow and gradual, compared to that of the 19th. Why is that? (I’ll give you a minute….figure it out? Of course you did.) The vertical scale isn’t the same in the 36-hour graphs above. Gah!

So, accounting for that difference, the overall rate of disk usage is about the same when Spotlight indexing is off, regardless of whether indexing Mail/Messages is turned on. That actually makes me feel better (I was worried that I’d accidentally changed something else that was affecting my data).

And yet…it’s still happening?

I was all set to clear the disk and start my last round of data gathering on the 26th. Then I discovered that data had returned to SpotlightKnowledgeEvents - over 3 GB of it. Shoot. I thought I’d stopped this. On the other hand, this now helps explain some of the graph shapes.

Here’s an extended graph, from Feb 23rd through the morning of the 26th. It shows regular jumps (with the mysterious spikes), until I clear everything and change the settings on 2/24. After that, we see a slow gradual rise. But at 4:00 this morning, there’s another big jump. This jump is similar to the jumps on the left half of the graph (minus the spikes), when Spotlight and Mail/Message indexing were enabled. A sudden drop in the graph is observed just after 6 am, when automatic thinning of snapshots occurred.

Here are the journal files in the Knowledge Events folder (note the five files created in the 4:00 hour, and especially, their size):

 19621941 Feb 25 04:14 [...]_101.journal
  5036176 Feb 25 10:15 [...]_102.journal
  2353146 Feb 25 12:32 [...]_103.journal
  1657257 Feb 25 14:31 [...]_104.journal
   320151 Feb 25 14:40 [...]_105.journal
  3716297 Feb 25 14:58 [...]_106.journal
 17139999 Feb 25 22:13 [...]_107.journal
387680291 Feb 26 04:17 [...]_108.journal
373296378 Feb 26 04:22 [...]_109.journal
599060522 Feb 26 04:26 [...]_110.journal
786612789 Feb 26 04:31 [...]_111.journal
689151006 Feb 26 04:34 [...]_112.journal
   351422 Feb 26 06:05 [...]_113.journal
   546614 Feb 26 06:32 [...]_114.journal
   434496 Feb 26 06:47 [...]_115.journal
  1914145 Feb 26 09:31 [...]_116.journal
    42220 Feb 26 09:32 [...]_117.journal

(I redacted most of the filename for readability. The full names look like this: skg_events_5DAB75C0-B6AE-4644-8F7D-60004EF98588_16777230_1207784874_journalAttr_117.journal)

Let’s compare this against the afternoon of 2/21, when Spotlight Disk and Mail/Message indexing were both enabled. Again, we’re seeing regular spikes, occasionally concurrent with an overall jump in usage, all mixed in with the regular gradual rise of usage. The mouse hover-over calls out a specific measurement just before such a (non-spiky) jump. This allows us to see the (nearly precise) time that the jump occurred: 4:14 pm.

I chose this stretch to graph because it still exists on the backup drive (it’s over a week ago, so I can only look at daily backups, not hourly). Here are the journal files that correspond to that jump (again, note the large files created in the 4:00 hour):

    731084 Feb 21 05:54 [...]_69.journal
    422916 Feb 21 06:33 [...]_70.journal
    983133 Feb 21 10:40 [...]_71.journal
   2950303 Feb 21 15:29 [...]_72.journal
 656602893 Feb 21 16:15 [...]_73.journal
1019889249 Feb 21 16:19 [...]_74.journal
 890047377 Feb 21 16:22 [...]_75.journal
 279774911 Feb 21 16:25 [...]_76.journal
 298012408 Feb 21 22:15 [...]_77.journal
 262945814 Feb 22 04:15 [...]_78.journal

These big jumps definitely seem to correspond to SpotlightKeyEvent journal files being collected / built / not-thrown-away. It’s the clutter, visualized. (Which, looking around my desk, I don’t usually need to have my clutter visualized, but that’s a different intractable problem.) I don’t have files corresponding to the jumps at 9 pm, 3 am, etc., because those would’ve been deleted as I cleared space for the next round of data collection, and the hourly backups that contained those files have been purged. Still, I’m reasonably confident that these jumps represent the SKE data growth.

I wondered if this might also explain the “catching up” jumps I saw earlier (Spotlight off, 2/19 about 10:00 am). Looking closer at the graph, that jump is about 2%. Out of a 233 gig (usable space) drive that’s 4.6 gigabytes. But in my backups from that time frame, I only have 33 megs of data (2.7 meg added in the 10 am hour). I still think those are some weird artifact for how snapshots work, esp. if early ones are purged but later ones remain.

Can I draw any conclusions?

I had one key question that I tried to address here: Does the disk fill up badly with Spotlight on its own, or only when both Spotlight and Mail/Message results are enabled?

Can I answer that question? Let’s review what we have so far:

At first glance, it looks like the answer is clear – turning off “Show Results for Mail & Messages” is enough to stop SpotlightKnowledgeEvents from filling up the disk. Except for that pesky 3 gig jump that happened before I started the final test. That one…kinda messes it all up.

Turns Out™, science is hard.

Here’s a graph of the full test, from the morning of February 19 until the afternoon of the 27th. You can see the four test periods (Drive indexing on & Mail results on, etc), as well as a brief period in the middle where I’d disabled Mail results but left Drive indexing enabled. (Hey, I tried to be methodical, but things slip). The “Off / On and “Off / Off” periods seem pretty similar, except for the 3 gig jump the next morning. The “On / On” test shows very high utilization in the 90-95% range. And “On / Off” shows a continuous rise, but steeper than the two tests where drive indexing was off.

It should also be noted that I was inconsistent about snapshot thinning, so that’s probably affected the data. And I haven’t been tracking the size of the Spotlight index on the drive, separate from the personal spotlight data in ~/Library/Metadata. That drive index hit 14 GB today, which seems a little large, to be honest, but maybe that’s just what it is.

So – do I have any conclusions?

I’m still confident that this is the bug that’s been eating my drive, but clearly Spotlight drive indexing, and the accumulation of Time Machine snapshots, also have a big effect. Plus, there’s that pesky unexpected jump in SpotlightKnowledgeEvents after 40 hours of minimal data accumulation (I was so certain I’d found a solution!).

So my conclusion is: I dunno. More testing is required, and in the end, the solution may simply come down to manually deleting this SKE folder from time to time, unless the bug gets fixed.

What’s next?

I think I’ll leave things alone for another evening (see if I get another jump). Then:

• Upgrade to 15.3.1: I’ll need to do that eventually, so I might as well. Then I’ll go back to On / On, and see what things look like. It’s possible that the 15.3.1 update will fix the bug, and so my testing won’t necessarily match up with what I’ve done here, but we’ve already seen that my rigor has … limits… So I don’t mind. I mean, as long as things get stable, right?
• Then maybe after a couple days (early next week), I’ll go back to Disk On, Mail/Messages Off, and expand spotlight indexing:
  • Does enabling indexing on external drives change anything? (I’d hope not)
  • What about indexing of the backup drive? Maybe it won’t affect SpotlightKnowledgeEvents, but it might affect the Time Machine snapshots. It’s worth trying for a day or two, anyway.

Also, I think I’m going to want to expand my crontab-triggered snapshot thinning. It’s clear that the OS is happy letting the drive get all the way up to 95% full before it kicks in. Which would be fine, if I didn’t also have lots of variability (from this bug, or other indexing, or just normal use).

If Other Things push the drive close to 95% on a regular basis, I may not be aware of any problems until things start to crash again (because there’d be no snapshots left to thin out). This is what I’ve been living with for the last 5 months, and it was damned near untenable.

How exactly I can do that (what command to use, how it maintains a high-water mark, how often to do it, etc.) is going to require yet more experimentation, as the documentation for tmutil isn’t terribly helpful.

Epilogue: What the heck are these spikes?

These continue to mystify me, though I haven’t looked too closely at them. They only last about 15-20 minutes (it’s hard to gauge exactly, even in the live Home Assistant interface). Could they be from temporary files built when the snapshot is created, then immediately deleted? I don’t think that’s how APFS Snapshots work, but it could still be related to Time Machine somehow.

Is there any easy way to figure these out? I looked into fs_usage and lsof, which both provide a helluva lot of information… so much that it’s hard to parse through. If these spikes return in full force in future testing, maybe I’ll try a full-disk fs_usage monitor for an hour, to try and capture file system writes during the spikes. That should, hopefully, at least tell me the offending process?

In the end, I’ve taken a few steps forward: I may have found a way to mostly mitigate the SKE growth, and I’ve also seen more clearly the impact of overall spotlight drive indexing. I’ve also taken some steps backward: the mitigation wasn’t reliable (that 4:00 jump), and I’ve raised new questions (those spikes).

I guess that’s just how research works, eh?