is my approach right to do it in UE4? if it is wrong, what approach should I try in UE4?
SetGlobalBoundShaderState is for global shaders only - those which don’t depend on a mesh or a material.
Tessellation as implemented in UE4 is a material feature, the material requests a tessellation amount and displacements. As a result, it’s the responsibility of the material’s pass-specific shaders (FMeshMaterialShaderTypes, like TBasePassHS) to implement the tessellation parts.
What you are trying to do requires support for a different primitive type - lines. I think you will need to create a new vertex factory based off of GPUSkinVertexFactory for this. The vertex factory will need to declare it’s primitive type (triangle or line) and the tessellation shaders like FlatTessellation.usf need to pick the right topology based on that.
I’ll be blunt, this is going to be a lot of difficult work, and probably not something to tackle if you are new to UE4. A much easier implementation is to just have very high poly hair with triangle strips, with discreet LODs to lower polycount when you are further.
can I get information about top view of UE4 technical graphics rendering, especially in shaders (from UE4 to USF) processing?
It’s hard for me to answer such a general question like this, as it’s such a wide question. But I can answer specific questions much better.
Short summary - a UMaterial compiles many shaders to implement the various rendering features. Vertex factory types are used to implement different mesh representation, for example particle sprites or rigid meshes or skinned meshes. Mesh material shader types are used to implement forward passes, for example the base pass or a shadow depth pass. Vertex factories are responsible for consuming the vertex inputs from the vertex buffer, and turning it into general vertex data through interface functions like VertexFactoryGetWorldPosition. That way the mesh material shaders can access world position without knowing what vertex factory is being used, they are decoupled.
how is the good practice of debugging shaders (the USF parts)?
I generally will output the value I’m investigating to the screen, and recompile shaders as I make edits to verify the results. When editing global shaders, ctrl+shift+. will recompile all shaders dependent on the modified files quickly. However for material shaders it will take a long time, it’s much faster to just iterate on one material by clicking the Apply button in the material editor after saving the shader changes.