Writing Custom Cel-Shading & Rim Lighting in HLSL for Unreal Engine 5

Getting a clean anime look in Unreal means stepping outside the default physically based lighting. In this tutorial we replace the standard diffuse response with a custom toon ramp and add a rim light you can actually control.

Why default PBR fights stylized art

Unreal’s deferred renderer is built for physical correctness. That’s great for realism, but it causes problems for stylized projects:

• Muddy shadows. PBR diffuse fades smoothly across curved surfaces, when stylized characters need sharp, deliberate shade lines.
• Uncontrollable rim light. Fresnel depends on roughness rather than the view and light angle, so rim light bleeds into places you don’t want it.
• Fragile outlines. Post-process outlines alias at a distance and can’t be tinted per material.

The ramp formula

Instead of a hard threshold (which crawls during camera motion), we use a smoothstep with a small feather around the threshold:

Ramp(N, L) = smoothstep(threshold - feather, threshold + feather, dot(N, L))

threshold sets where light turns to shadow, and feather is a tiny blur that keeps the edge anti-aliased.

The HLSL

Create a Custom material expression and add inputs for WorldNormal, LightDir, ViewDir, and the parameters below. Paste this into the Code field:

float3 N = normalize(WorldNormal);
float3 L = normalize(LightDir);
float3 V = normalize(ViewDir);

// 1. Half-Lambert diffuse
float NdotL = dot(N, L);
float halfLambert = NdotL * 0.5 + 0.5;

// 2. Two-step ramp with a small anti-aliasing feather
float feather = max(0.001, FeatherRadius);
float tone1 = smoothstep(Threshold1 - feather, Threshold1 + feather, halfLambert);
float tone2 = smoothstep(Threshold2 - feather, Threshold2 + feather, halfLambert);

// 3. Blend deep shadow → shadow → lit
float3 baseTone = lerp(DeepShadowColor.rgb, ShadowColor.rgb, tone1);
float3 litTone  = lerp(baseTone, AlbedoColor.rgb, tone2);

// 4. Rim light, masked to the lit side
float rimDot = 1.0 - saturate(dot(N, V));
float rimMask = pow(rimDot, RimExponent) * saturate(NdotL + RimLightWrap);
float3 finalRim = rimMask * RimColor.rgb * RimIntensity;

return float4(litTone + finalRim, 1.0);

Parameters

Parameter	Range	Start with	What it does
Threshold1	0–1	0.32	Separates deep shadow from mid shadow.
Threshold2	0–1	0.58	The main light/shadow line.
FeatherRadius	0–0.1	0.015	Softens the edge to prevent stair-stepping.
RimExponent	1–16	4.5	Higher values give a thinner rim.

That’s the whole shader. From here you can swap the lerp calls for a ramp texture lookup if you want hand-painted color bands.