[Python] Wrong color space of infered image in Unity

I'm having a hard time infering an image from a camera in Unity to an ONNX using a render Texture. I realized that the output from the ONNX inferenced with Barracuda is significantly different from the output I get when I infer the same image (after capturing) in python to the same ONNX.

The goal is to only use Unity for inferencing the ONNX, but that's not working right now, and I don't know how to fix this.

In Unity:

My render texture has the Color format R8G8B8A8_SRGB, height and widht align with the script below:

script:

public class LaneONNXRunner : MonoBehaviour
{
[Header("ONNX Model")]
public NNModel modelAsset;

[Header("Input Settings")]
public Camera laneCamera;
public RenderTexture renderTexture;
public int modelInputWidth = 1280;
public int modelInputHeight = 720;

[Header("Async Settings")]
[Tooltip("How many inference steps to run per frame. Lower = smoother UI, slower inference.")]
public int stepsPerFrame = 1;

private Model runtimeModel;
private IWorker worker;
private Texture2D inputTexture;

void Start()
{
runtimeModel = ModelLoader.Load(modelAsset);
worker = WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, runtimeModel);

inputTexture = new Texture2D(modelInputWidth, modelInputHeight, TextureFormat.RGBA32, false);
//renderTexture = new RenderTexture(modelInputWidth, modelInputHeight, 24, RenderTextureFormat.ARGB32);
}

void Update()
{
if (Input.GetKeyDown(KeyCode.I))
{
StartCoroutine(RunAsyncInference());
}

}
...
private void SaveInputAsJPG(Texture2D tex)
{
byte[] jpgBytes = tex.EncodeToJPG(95); // 0-100 quality, 95 is a good default
string saveDir = Application.dataPath + "/../../liveCaptures3";
if (!Directory.Exists(saveDir))
Directory.CreateDirectory(saveDir);

string path = saveDir + $"/SavedONNXInput_{System.DateTime.Now:dd_HHmmss_f}.jpg";
System.IO.File.WriteAllBytes(path, jpgBytes);
Debug.Log($"[Saved] JPEG input image saved to: {path}");
}
...

IEnumerator RunAsyncInference()
{
// Step 1: Capture image from RenderTexture to Texture2D
laneCamera.Render(); // Ensures camera writes to RenderTexture
RenderTexture.active = renderTexture;
inputTexture.ReadPixels(new Rect(0, 0, modelInputWidth, modelInputHeight), 0, 0);
inputTexture.Apply();
SaveInputAsJPG(inputTexture); // Add this

RenderTexture.active = null;

// 1.2. Encode to JPG and reload to simulate the JPG decoding (like in Python)
byte[] jpgBytes = inputTexture.EncodeToJPG(95);
//Texture2D jpgTex = new Texture2D(2, 2); // size wil be overwritten
//jpgTex.LoadImage(jpgBytes);
Texture2D jpgTex = new Texture2D(2, 2, TextureFormat.RGBA32, false, false);
jpgTex.LoadImage(jpgBytes);

// Step 2: Convert to Tensor
//Color[] pixels = inputTexture.GetPixels();
Color[] pixels = jpgTex.GetPixels();
float[] floatValues = new float[pixels.Length * 3];
for (int i = 0; i < pixels.Length; i++)
{
Color c = pixels;

floatValues[i * 3 + 0] = c.r / 1.0f;
floatValues[i * 3 + 1] = c.g / 1.0f;
floatValues[i * 3 + 2] = c.b / 1.0f;

}

string preview = "";
for (int i = 0; i < 12; i++)
preview += floatValues.ToString("F10") + ", ";
Debug.Log("Unity Input Preview: " + preview);

...



This logs for example the following to the console:

Unity Input Preview: 0.1960784000, 0.1960784000, 0.1960784000, 0.1960784000, 0.1960784000, 0.1960784000, 0.1960784000, 0.1960784000, 0.1960784000, 0.1960784000, 0.1960784000, 0.1960784000,


Then I use the same image I just captured in infer it in Python, to see if there is any significant difference:

import onnxruntime as ort
import numpy as np
from PIL import Image
import torchvision.transforms as T

img = Image.open("inference_input/tteesst.jpg").convert("RGB")

transform = T.Compose([T.Resize((720, 1280)), T.ToTensor()])
input_tensor = transform(img).unsqueeze(0).numpy()

print(input_tensor.flatten()[:12])

...


This prints

[0.53333336 0.53333336 0.53333336 0.53333336 0.53333336 0.53333336
0.5372549 0.5372549 0.53333336 0.53333336 0.53333336 0.53333336]



Notably, the inference done in the python script leads to much better prediction, than the one in the Unity script. But the Unity scipt's inference is not so bad that I would assume that I flipped the image over any axis (including diagonals).

I'm pretty sure it has something to do with the color space in Unity not matching the one expected by the model.

I have tried for hours to change the renderTexture, to convert the color space in Unity, but it seems I don't really understand what is happening, since nothing works.

I even added some lines to first save the input as JPG and then load it again, like in Python, but to no avail.

I'm totally lost here, and I don't seem to understand what's going on in the GetPixel() method in Unity vs. Image.open() / Image.open(...).convertin Python.

I'd be sooo glad if someone could help me.

Continue reading...