I am glad you found a fix.  As long as the foam retains its "springiness" you should be OK. 

Years ago we had a VW Rabbit whose plastic steering wheel center was a large piece of molded plastic serving as both column padding and the horn push button.  In extremely cold weather the plastic would shrink and the horn would honk itself (usually late at night to the joy of my neighbors).  Similar to your situation, I fixed the problem by removing the plastic and placing small squares of sticky-back firm weatherstrip tape in key locations.  The tape was still doing its job 15 years later when we finally sold the car.

Just to follow up - chased items in the steering column and didn't see anything amiss. Seemed like the issue was in the center button itself- the upper plate contact ring and the lower plate/contact ring. So first attempt was the build up the cup area in the upper plate a bit (brass dress/cup washer fit perfectly) to see if extra tension from the spring would hold the plates apart more firmly. It worked and would have been great except the center button assembly is attached by only one screw and the xtra tension pushed the opposing side up to an unacceptable level.  The second approach was to wedge a couple very small/tiny pieces of foam (modern headliner material) across from each other between the 2 plates.  This seems to keep the two plates from randomly touching but since it only insulated the small areas on each side still allows the horn to function when mashed. We'll see how it holds up but for the moment ...........  Thanks for the suggestions as always.  
Kim

Yes, the Mk1 horn is always "hot" and works when the ground connection is completed.  It is very reasonable to look for where on the steering column the "unwanted" path to ground is being made which is causing the horn to honk.  Things to check are as mentioned above, the brass slip ring on the column, the spring loaded wire and plastic insulator "inside" the brass slip ring, the spring loaded contact in the center of the steering column, the wiper that rides on the slip ring (on the turn signal switch), and the horn push button itself.  Though less likely on the Mini than other British cars, wear in the steering column bushes can allow excessive motion that allows parts to short together.

Doesn’t the horn work when earthed??  I would checking the wire in the column for a bare spot/damaged insulation.

Cheers, Ian 

Check that the brass collar isn't contacting the indicator cancelling screw.

Something is cross connecting the horn circuit in your system..
 this may be why they converted away from the Steering wheel button to a separate system.. A hidden problem.

 I would be leary of the wire that comes up the center of the column, that connects to the horn push.
Internally it could be Knicked and just barely grounding / touching somwhere at the right way to activate that horn.

Its a fun job!

Removoe and   Dissassemble the column.   then you are left with the shaft.
then you have to remove the brass ring from the Plastic isolators

https://www.7ent.com/products/horn-push-contact-assembly-mk1-str0069.html


Then the plastic isolator splits in half from the steering shaft.
 LIKELY the column parts are aged and brittle , falling apart causing issues.


BUT i would also search for it grounding  somewhere else in the system

MK1 Morris- When the car arrived it had a toggle switch to control horn. I installed new turn signal switch, cleaned the contacts and redirected the horn back to the steering wheel (big original) center.  Works too well- if I hit a bump or turn too vigorously or otherwise look at it cross the horn pops off.  Entertaining the first time but not so much at a stop light with a big 4x4 looking down.  Any tips on making it behave?  
Thanks
Kim