How to fix cloth physics doesn't align properly?

So I’m trying to give physics to my character’s clothes using the built-in Cloth Paint. But after I apply the cloth asset to the cloth material on my character, the cloth suddenly out of align, but strangely enough the cloth still follows the bone.

After looking somewhere between UE forums and answerhub, many people pointed out that this maybe the problem with the approximated size of the skeletal mesh. However, this only occurs when I apply a cloth asset to a material. Unapplied, my character’s approx size is 161x45x169. I’ve already apply any kind transformations to the mesh in Blender and it’s already in rest pose while exporting to FBX. I’ve also already set FBX scaling to 1.0 and units to metric (if that matters). The cloth meshes is one armature with the character. But even after those settings applied when exporting the mesh to FBX and reimport it in UE, this problem doesn’t resolved at all.

For the physics asset, I made a separate physics asset specifically for the clothes to collide as accurate to the skin, using a version of my character with clothes and hairs deleted, remaining only her bare skin. The remaining mesh was exported after I exported the character with clothes and hairs included, so no further modifications were made. I created the physics asset using Single Convex Hull per bone (except the root) to fit the collisions to the skin.

Weird thing is, I once rigged the same character in Mixamo, after I strip out the armature, and I can apply the clothing asset without similar issue, at the cost of broken finger weights. The weights took many time to fix manually, and unfortunately that’s the only way, so I discarded the skeletal mesh rigged by Mixamo. I don’t remember the approx size for that mesh since I wasn’t paying attention to that.

I already checked both physics, the physics for ragdoll and the physics for clothes to fit the skin, and there’s seems like no sign of mesh misalignment as far as I can tell.

So my question is: With those details I’ve written, is there any workaround to fix this clothing physics issue?

PS: I’m using AMD GPU right now, so I don’t think Nvidia tools / SDK will work properly.