Yes, that is the premise. But it isn’t flawed. If you spend 5-10 minutes running to a gathering spot or running a route and you get killed then that is 5-10 minutes you aren’t gathering and that you have most likely lost the route completely. That is 5-10 minutes more solid gathering that a luck geared, food buffed PvE gatherer is making progress. If someone else is on the route then they just skip ahead while that player is harvesting and it is still not a total loss.
The assumption that PvP gathering is generally more efficient relies on the flagged player being uninterrupted. And that is not the experience the OP described. They described constant ganking and conflict. Which is a major hit to efficiency and diminishes the value of the buffs.
And you need to remember that every obstacle with gathering a PvE player has (a “swarm of gatherers” in your example) also applies to and compounds with PvP specific obstacles. For example, you mention being forced to a new location. A PvE player just moves. A PvP player could also be ganked in that process and have to start over back in town which is again a net loss in efficiency.
A PvE player can purchase that 2000 luck food, has the freedom to scout an uncontested location and achieve 30 minutes of uninterrupted fully buffed gathering. A PvP player cannot.
Every second a PvP player has to spend in combat with another player, hiding, running from spawn or carefully relocating is a gathering deficit versus unflagged players that a passive buff reduces over time.

