Is it a back or a core exercise?

Mostly core with a back stimulus. If back size is the goal, a supported row lets you use far more load.

How do I stop rotating?

Feet wider, glutes squeezed, and a slower tempo. Rotation is the exercise telling you the load is too heavy.

In a plank on two dumbbells, you row one at a time while keeping the hips square. It is a back exercise with an anti-rotation core demand bolted on.

Primary: Lats, Upper back, Core. Secondary: Biceps, Chest (pectorals), Glutes.

Set two hexagonal dumbbells shoulder-width apart and take a plank position on them.
Widen the feet for a stable base and brace the trunk hard.
Row one dumbbell to the ribs while keeping both hips level.
Lower it under control and repeat on the other side.

Hips rotating toward the rowing arm. Widen the feet, slow down, and lighten the load until the hips stay square.
Round dumbbells rolling under the hands. Use hexagonal dumbbells or kettlebells; round ones are genuinely unsafe here.

6 to 10 reps a side with moderate loads. Judge it by whether the hips stay level, not by the dumbbell size.