Unable to decompile shaders that write to depth

[RaphaelK12]

Hello, I'm having errors in the pixel shaders that write depth.

Unhandled exception. System.Collections.Generic.KeyNotFoundException: The given key 'DepthOut0' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at HlslDecompiler.Hlsl.RegisterState.GetRegisterName(RegisterKey registerKey) in F:\Dev\HlslDecompiler\Hlsl\Compiler\RegisterState.cs:line 168
   at HlslDecompiler.Hlsl.RegisterState.GetDestinationName(Instruction instruction) in F:\Dev\HlslDecompiler\Hlsl\Compiler\RegisterState.cs:line 48
   at HlslDecompiler.HlslWriter.GetDestinationName(Instruction instruction) in F:\Dev\HlslDecompiler\Hlsl\HlslWriter.cs:line 45
   at HlslDecompiler.HlslSimpleWriter.WriteInstruction(D3D9Instruction instruction) in F:\Dev\HlslDecompiler\Hlsl\HlslSimpleWriter.cs:line 230
   at HlslDecompiler.HlslSimpleWriter.WriteMethodBody() in F:\Dev\HlslDecompiler\Hlsl\HlslSimpleWriter.cs:line 28
   at HlslDecompiler.HlslWriter.Write(String hlslFilename) in F:\Dev\HlslDecompiler\Hlsl\HlslWriter.cs:line 124
   at HlslDecompiler.Program.ReadRgxa(String baseFilename, FileStream inputStream, Boolean doAstAnalysis) in F:\Dev\HlslDecompiler\Program.cs:line 91
   at HlslDecompiler.Program.Main(String[] args) in F:\Dev\HlslDecompiler\Program.cs:line 30

[RaphaelK12]

I made some changes to the program to get around the exceptions, I got something semi-functional.

I obtained the following generated HLSL:

sampler depthSourceSampler;
float4 gooDeferredLightScreenSize;

float4 main(float2 vface : vFace) : COLOR
{
	float4 o;

	float4 r0;
	o = float4(0, 0, 0, 1);
	r0.xy = 0.51 + vFace.xy;
	r0.xy = r0.xy * gooDeferredLightScreenSize.zw;
	r0 = tex2D(depthSourceSampler, r0);
	oDepth0 = r0.x;

	return o;
}

The expected HLSL would be:

sampler depthSourceSampler : register(ps, s0);
float4 gooDeferredLightScreenSize: register(c66);

struct PS_OUT
{
    float4 Color : COLOR;
    float  Depth  : DEPTH;
};

PS_OUT main(float2 vpos : vPos)
{
	PS_OUT o;

	float4 r0;
	o.Color = float4(0, 0, 0, 1);
	r0.xy = 0.51 + vpos.xy;
	r0.xy = r0.xy * gooDeferredLightScreenSize.zw;
	r0 = tex2D(depthSourceSampler, r0);
	o.Depth = r0.x;

	return o;
}

I also changed it from line 72 of Program.cs to:

string outFilename;
if (shader.Type == ShaderType.Vertex)
{
    if (!Directory.Exists($"fx\\{baseFilename}\\"))
    {
        Directory.CreateDirectory($"fx\\{baseFilename}\\");
    }

    outFilename = $"fx\\{baseFilename}\\{baseFilename}VS{ivs}";
    ivs++;
}
else
{
    if (!Directory.Exists($"fx\\{baseFilename}\\"))
    {
        Directory.CreateDirectory($"fx\\{baseFilename}\\");
    }

    outFilename = $"fx\\{baseFilename}\\{baseFilename}PS{ips+1}";
    ips++;
}
Console.WriteLine(outFilename);

I made these changes to make the shaders more organized in subfolders and follow the RSE model where the pixel shader starts at 1.

I would like to have a fully functional decompiler for hlsl, I could edit it myself if I had time and knew more about C#, but unfortunately my time is short, I don't really understand how C# works and this project is a bit complex for me .

But anyway, awesome project, it helps me a lot in not having to edit the assembly directly and being able to use the facilities of a higher level language.

[AndresTraks]

Hi, I added support for depth output here: c088655

Unfortunately I don't have much time to work on all of the corner cases either, but I can probably fix most of the issues that are reported. I'm glad the project is useful for you!

I'll think about the subfolders solution later.

[RaphaelK12]

It's working great for depth and also vFace has been fixed for vpos/VPOS, great progress.

Now there are only 2 more errors due to unimplemented operations.

Compiler: Microsoft (R) HLSL Shader Compiler 9.29.952.3111
Shader model: ps_3_0
Unhandled exception. System.NotImplementedException: The method or operation is not implemented.
   at HlslDecompiler.Hlsl.RegisterState.GetSourceConstantName(D3D9Instruction instruction, Int32 srcIndex) in F:\Dev\HlslDecompiler-master\Hlsl\Compiler\RegisterState.cs:line 672
   at HlslDecompiler.Hlsl.RegisterState.GetSourceName(D3D9Instruction instruction, Int32 srcIndex) in F:\Dev\HlslDecompiler-master\Hlsl\Compiler\RegisterState.cs:line 69
   at HlslDecompiler.HlslWriter.GetSourceName(Instruction instruction, Int32 srcIndex) in F:\Dev\HlslDecompiler-master\Hlsl\HlslWriter.cs:line 54
   at HlslDecompiler.HlslSimpleWriter.WriteInstruction(D3D9Instruction instruction) in F:\Dev\HlslDecompiler-master\Hlsl\HlslSimpleWriter.cs:line 106
   at HlslDecompiler.HlslSimpleWriter.WriteMethodBody() in F:\Dev\HlslDecompiler-master\Hlsl\HlslSimpleWriter.cs:line 28
   at HlslDecompiler.HlslWriter.Write(String hlslFilename) in F:\Dev\HlslDecompiler-master\Hlsl\HlslWriter.cs:line 124
   at HlslDecompiler.Program.ReadRgxa(String baseFilename, FileStream inputStream, Boolean doAstAnalysis) in F:\Dev\HlslDecompiler-master\Program.cs:line 108
   at HlslDecompiler.Program.Main(String[] args) in F:\Dev\HlslDecompiler-master\Program.cs:line 30

Compiler: Microsoft (R) HLSL Shader Compiler 9.26.952.2844
Shader model: ps_3_0
Unhandled exception. System.NotImplementedException: The method or operation is not implemented.
   at HlslDecompiler.DirectXShaderModel.RgxaReader.ReadShaderStruct(Int32 numInputs) in F:\Dev\HlslDecompiler-master\DirectXShaderModel\Reader\RgxaReader.cs:line 70
   at HlslDecompiler.DirectXShaderModel.RgxaReader.ReadShader() in F:\Dev\HlslDecompiler-master\DirectXShaderModel\Reader\RgxaReader.cs:line 171
   at HlslDecompiler.Program.ReadRgxa(String baseFilename, FileStream inputStream, Boolean doAstAnalysis) in F:\Dev\HlslDecompiler-master\Program.cs:line 66
   at HlslDecompiler.Program.Main(String[] args) in F:\Dev\HlslDecompiler-master\Program.cs:line 30

It is also necessary to write the register index in the variables for a perfect decompilation, something that would enable a functional recompilation, I have seen many people who tried to recompile the code generated by their decompiler and the results were glitches, because if the constants and textures are in different registers of the originals, a lot of things won't work as they should. This isn't a big problem for me, but it could be for anyone who wants to decompile, recompile and have artifacts without knowing why.