The front tie bar bushings are pretty crushed.  I was planning on replacing those when I did my adjustable lower arms and tie rod installation.  I think maybe that's what my problem is.  While I can't say zero movement since everything will move with enough force, the ball joint movement was negligble.    
 

So you do see "some" play at the balljoints........you shouldn't see any.

And the rest of the movement I'm guessing is your wheel bearings....wider wheels/tyres will exacerbate the the wear.

How did the front of the tie bars look?  ( where they attach to subframe )

Ok, I had a chance to jack up the car and observe wheel play.  Both front wheels move a little when I push/pull vertically and laterally, but it seems the movement is at the hub, or brake drums, not the ball joints.  I don't see too much play at the ball joints.  What could this mean?

Last week I installed new wider wheels and tires, and have noticed that the wobbling while driving has become much less, but I'm thinking maybe the wider setup is just hiding the problem a little better. 

Rick: when you say "pulling" and "pull" I think you mean "pushing" and "push". (Unless of course you are driving in reverse... or is it because you have an Aussie car where everything is backwards?)

"Worrisome" - I agree!

To add to the tie rod discussion, here's what Kieth Calver says:

"Bush-wise there’s been an epidemic of options produced in all manner of ‘superior’ materials, most of which work satisfactorily. Harder varieties need regular inspection and replacement as they tend to ovalise/split after relatively low mileage on daily drivers. I advise running a hard bush on the front/outer side of the tie-rod mounting bracket, a standard/more compliant one on the rear/inside. The harder bush gives greater stability under braking, the softer one a longer ‘life’ through compliancey. An interesting consideration is that the very hard bushes resist body-roll, thus acting as an in-built anti-roll bar and therefore assist in cornering. This has merit, though I’ve never tried to quantify it. Mounting bracket support straps should be used when very hard bushes are fitted. I’ve seen these brackets bend when the car’s used in anger!"  

//www.minimania.com/SUSPENSION___Basic_Priorities_1083

Other less likely possibilities: alignment, badly worn/loose shocks and engine mounts.

Very worrisome...and I second the jacking up and checking each front wheel separately for play.  With power down, the front drive is pushing on the tie-rods/bushes/ears.  When you lift the throttle the "push" is gone and this is when the worn bits show their problems. (Edited - thanks to Dan...too much push me pull you with the grandkids)

Hold the wheels top and bottom (vertical) and see if there is any play when you push/pull/in/out.  Bearings & ball joint issues could be indicated and wiggly pin bushes on inboard end of lower arms.

Hold the wheels front to back (horizontal) and push/pull/in/out see if there is any play...especially in the steering rack/rack end or arm.

While there may be a cascading of several wear issues, I'm inclined to think that your tie-rod bushes are shot and I would inspect the ears on the front subframe for any cracks/bending which may be a result.

When I have modified front subframes with reinforcing stitch welds I have also gusseted the "ears" up front...especially with a more powerful motor to be installed.

 jwakil is it a steering direction change or just vibration?  Worn lower arm bushings have caused the former for me in the past.

And +1 for the advice from Spitz and Gromit. People have gone off the road and into the center divider when some of these parts let go.

I'm not an expert by any means....but that sounds a little scary.  Perhaps  the wheel caster is off

I'd lift the front up and check things out...grab each wheel and check bearing and ball joints for play.
Check also where the tie rods mount to the front of the subframe....you don't want that nut to be loose!...and check that those rubbers haven't deteriorated away.

SOunds like worn bushings or ball joints,  Best get that checked out or just don't let off on the gas  Actually that sounds pretty serious. you can jack it up and try wiggling it front to back, but you would be better off taking it to a shop.... don't wait either!

I'm trying to figure out what is causing front end wobbling/shaking on my 82 mini when I let off the gas at high speeds.  Even before I apply brakes the front end and steering wheel shake somewhat.  It doesn't happen if I'm accelerating or if the wheels are driving (engine under load) while at the same speed.   I have drum brakes, and I believe the wheel balance is ok since in normal cruising there is little to no vibration at any speed.