In the setting that you describe, the x and y components have no overall effect at all.
If you have multiple forces acting upon a (point) body, the result is exactly the same that a single force would have, if the single force is the vector sum of all the individual forces.
As your hole goes through the center of Earth, the mass distribution around this hole is perfectly symmetric in the xy plane, so all gravity force components along x or y will be compensated by opposite ones. The only asymmetry with regard to the falling object is in the z direction, so here the different gravity forces add up to something that is zero only at the center of Earth. And vector math tells us that, for the addition of vectors, we can treat the x, y, and z components individually, so whatever happens in x or y has no influence on z.