Chat-assistant privacy has a simple shape. You choose what to type or attach, and the question is what happens to it afterwards. The decision point is the moment you press send, and you can see exactly what you handed over.
Agent privacy has no such moment. You describe an outcome file these, find that, update the other and the agent works out for itself what it needs to look at. The scope of what it reads is decided by the task, not by you, and it is decided after you have stopped paying attention.
You did not disclose the folder. You disclosed a job that happened to require the folder. Those feel the same when you type them and are not the same at all.
None of this is misbehaviour. An agent that refused to read a directory listing could not sort a directory. The reach is the capability. But it does mean the honest question to ask before granting an area is not “do I trust this with the thing I am asking about?” it is “do I trust this with everything in the place I am asking about?”
Why this lands hardest on files
Mail, calendar and notes tend to be uniform: your inbox is all correspondence, your calendar is all appointments. A folder is not uniform. Downloads in particular is where people keep the scan of a passport next to a takeaway receipt, and “Documents” accumulates decades of things you would never hand to anyone deliberately.
So the practical advice is duller than a privacy setting and more effective than one: make a folder for the task. Move the twelve invoices into it, point Muse at that, and the question of what else it read answers itself. It costs thirty seconds and it converts an incidental-access problem into an intended-disclosure one, which is the kind you already know how to reason about.
The same logic applies upward. Granting an area is a standing decision, not a per-task one so the area you grant should be the smallest one that makes your ordinary tasks work, not the largest one that makes every conceivable task work.