Display screens connected to cameras are significantly different from windows or mirrors. Looking through a window or mirror, we can focus on infinity, and we can move our vantage point to see a little different. With a display screen showing live camera footage, we see the field of view from a single point (the camera location) and our focus is on the screen.
Would it be possible to develop a screen + camera that is much closer in behaviour to a window or a mirror? I think the screen and camera would need to have the same surface area. Both would be sensitive to direction, and when camera pixel $(C_i, C_j)$ on the camera receives a photon with frequency $\nu$ from angles $(C_\phi, C_\theta)$, the screen would send a corresponding photon at frequency $\nu$ from position $(S_i, S_j)$ to direction $(S_\phi, S_\theta)$, where $(S_\phi, S_\theta)$ are calculated from $(C_\phi, C_\theta)$ mimicking either window- or mirror-like behaviour.
Is such a device theoretically possible? If yes, would such a device be technically feasible today? If yes, has there been any serious work on such devices? If it's theoretically possible but not feasible today, what would need to be developed before such devices may be on the horizon?
It should have a wide range of applications in telepresence, augmented reality, automotive engineering, and surely a lot of other fields.