do all those COPY lines result in intermediate layers?

Could you maybe modify stage 2 to:

    FROM scratch as stage2
    
    COPY ...
    COPY ...
    ...

and finally at the end:

  FROM SCRATCH

  COPY --from=stage2 / /