You've got a whole lot of fail going on with your connections. You need to read the manual for your mixer. It's worth it.
http://download.yamaha.com/api/asset/fi ... et_id=8050
You can't use the XLR and line input jacks on the same channel at the same time.
You can't use the Insert jack the way you are. Insert is not an input. It's a monophonic send and return.
The KP3 is a stereo effect unit designed for an effects loop.
What you need to do:
Mic is connected to the XLR on channel 1.
Nothing else is hooked up to channel 1.
The mono effects send on the mixer needs to go to the KP3 input(s). It can go to the mic input by using a standard instrument/patch cable, 1/4" mono TS plug to 1/4" mono TS plug. Or it can go to the stereo line inputs on the KP3 by using a Y adapter cable or adapter, 1/4" mono TS plug to a pair of RCA. If you go to the mic input then you can control the input level with the Mic Trim knob on the front of the KP3. You flip the mic input switch on the KP3 to pick which input you used. The input level knob needs to be adjusted so that the input level LED is mostly green, occasionally peaking with a yellow, and never red (overloaded).
Then you need to hook the KP3 stereo line outputs to the Effect Return L+R jacks on the mixer. So you need a pair of RCA to 1/4" mono TS plug cables, or a single stereo cable that has 2 RCA and 2 1/4" mono plugs. You can adjust the amount of the KP3 effect by using the Effect Return knob on the mixer. The connection switch needs to be set to SEND on the KP3.