I came across this interesting piece: C.L. Swartz, “Libel,” in Charles T. Sprading, ed., Liberty and the Great Libertarians: An Anthology on Liberty: A Hand-book of Freedom (Los Angeles: The Golden Press, 1913), p. 526. It is short, so I reprint in full below. But he hits on the essential point: speaking words does not (normally) cause an invasion. Thus it cannot be penalized by law. (For more on this see Murray N. Rothbard, “Knowledge, True and False,” in The Ethics of Liberty (New York: New York University Press,  1998), Walter E. Block, “The Slanderer and Libeler,” in Defending the Undefendable” (Auburn, Al.: Mises Institute,  2018), Kinsella, “Causation and Aggression,” in Legal Foundations of a Free Society (Papinian Press, forthcoming 2023), and idem, “Defamation as a Type of Intellectual Property,” in A Passion for Justice: Essays in Honor of Walter Block (Addleton Press, forthcoming).)
C. L. Swartz
Under a rational conception of free speech there can be no such thing as libel, considered as an invasive act. Speech after all is not a complete act. An indispensable complement is the hearing of what is said. And even then the thing does not attain to the dignity of an act. An invasion must be an overt act. To determine an invasion, the consequences of the overt act must be considered. To say a thing, no matter how untrue, outside of the hearing of anyone, is, it is clear, of no consequence. There is no reason, therefore, to believe that the situation changes, in so far as the speaker is concerned, when the thing spoken is heard. And neither does the simple fact of its being heard alter the conditions. It is only when the hearer thinks or takes action that any person lied about can feel the effect of the lie. He could not be injured by it if it were not heard; he could not be injured by it if it were heard and not believed; he could not be injured by it if it were heard and believed if no action were taken by the person hearing and believing it. It is only when a person hears a lie, believes it, and then takes some action toward the person lied about that the latter can be injured. After the liar has told his lie, three things must take place before it can have any injurious effect, and these three things are in no wise connected with the liar. What, then, has the liar to do with it anyway?