The really quick and dirty solution: weld the chuck to the taper. That will ensure it won't come flying off. Then you can do light milling and start making parts for a stronger/better machine!
I wouldn't bother throwing a lot of money at this as drill presses are generally not rigid enough to make 'good' milling machines. Any welder should be able to put some large tac-welds (I'd do four 90 degrees apart) on that in about 5 minutes which would prevent it ever coming off (be careful about doing too much welding as you don't want to introduce too much heat into the part).