[4eyes] Updates from SIGGRAPH (Days 4 and 5)

[Pradeep Sen]

Hi everyone,

Here is my last update from Vancouver, this one for Days 4 and 5. Enjoy!


*PAPER TALKS*

*Rendering*

Vevoda et al. "Bayesian online regression for adaptive direct illumination
sampling"
This paper performs online learning of direct lighting to do importance
sampling of light
sources for path tracing techniques. This is similar to what we were
exploring when we
(Steve and I) first started the project with Pixar last summer.  Should we
have just written
that up and published it? Probably! It is good to observe when one becomes
so focused on
the "deep" problems that we miss the lower hanging fruit. We need to read
through this and
figure out what is still salvagable of that original work.


*AR/VR*

Li et al. "
Scene-Aware Audio for 360° Videos
"
I didn't go to this talk, but it looks interesting. The basic idea is that
standard audio does not
work well for 360 videos because it doesn't account with how people turn
their heads as they
are looking around. I.e., in traditional approaches, the sound always
follows the user, so that
as you rotate your head the sound follows.  In this approach, they code the
sound in such a
way so that it feels like it is located at that same physical location
regardless of how you turn
your head. I don't know much about sound, but it seems interesting.


*3D Reconstruction*

Hedman et al. "Instant 3D Photography"
Aligns multiple RGBD images and stitches them together to turn them into a
mesh so that
they can be viewed in VR. Very impressive results!

Whelan et al. "
Reconstructing Scenes with Mirror and Glass Surfaces
"

This paper addresses the problem of scanning glass/mirror surfaces for 3D
scanning. They
detect mirrors by the observation that if you can see yourself (i.e., the
camera) then you are
likely looking in a mirror.  So they put an "April Tag" on the camera that
can be seen in the
images and they use it to automatically recognize the mirrors and remove
them, to produce
nice results. (This was the paper that Chris Sweeney told us about.)

Wu et al. "
Full 3D Reconstruction of Transparent Objects
"

Proposes a way to 3D scan transparent objects
by finding
ray-ray correspondences entering
the
transparent object
from structured illumination and then optimizing the surface.

Lindell et al.
"Single-Photon 3D Imaging with Deep Sensor Fusion"
This method proposes a way to combine information from a low-resolution
low-sample count
time-of-flight imager with a high-res intensity image in order to produce
high fidelity depth maps.

Yin et al. "P2P-NET: Bidirectional Point Displacement Net for Shape
Transform"
This paper proposes to a deep learning network that moves points around to
form structures.
They do this with a bidirectional training, kind of like Cycle GAN.  I
think there could be
applications of these ideas to rendering,


*Relighting*

Xu et al. "
Deep Image-Based Relighting from Optimal Sparse Samples
"
This paper tries to solve the image-based relighting problem using deep
learning.
While
normally
light transport is measured independently from scene to scene (essentially
almost overfitting
to
each individual scene), they trained a system that would take five
captured images and then
learn
to interpolate to match a lot of lighting configurations.
Interesting that they generated
random objects
f
or their training
procedure simply by
deforming meshes randomly.




*Computational Photography*

Sitzmann et al. "
End-to-end Optimization of Optics and Image Processing for Achromatic
Extended Depth of Field and Super-resolution Imaging
"
Very nice idea which started with a question: is the relationship between
the camera optics
and
the point spread function (PSF) differentiable? If so, we can put the
optical design within
an
optimization to figure out how to construct a lens that would achieve a
particular effect.
For
example, they solve for the optical height map that would give them a sharp
image, and
they
found something similar to a Fresnel lens. They did end to end training
along with the
reconstruction algorithm, and were able to apply it to a couple of real
problems, such as
addressing all-in-focus imaging, etc.  Awesome!

Deschaintre et al. "Single-Image SVBRDF Capture with a Rendering-Aware Deep
Network"
This paper proposes to use a CNN that takes as input a single image and
outputs a spatially
varying BRDF. Specifically, their BRDF model consists of the surface
normals, as well as
spatially-varying diffuse, roughness, and specular terms.. One key is that
rather than
compare against the GT maps for training, they do their loss against their
rendering, which
requires a differentiable rendering system. Abhishek, you may want to look
at this paper as it
is related to what you are working on and we have discussed many similar
things.


*HPG*

Poms et al.
"Scanner: Efficient Video Analysis at Scale"
This paper proposes a
 software system for large-scale video
processing that allows the user
to
run image processing tasks on large sets of videos on heterogeneous
computing clusters.



*COURSES*
There was a pretty interesting course on Rendering and Machine Learning on
Thursday 2-
5pm. The course notes seem to have a lot of links to relevant literature,
so we should pore
through that carefully.  For example, Jan Novak talked about some new work
they just
released to arXiv that seems really related to the project Steve was
working on for the past
couple of years:

https://arxiv.org/pdf/1808.03856.pdf

They are using machine learning to learn an importance sampling function.
We should read
through it and discuss.



*OTHER*
As usual, there were also lots parties (Facebook had a party for the first
time, along with the
usuals NVIDIA, Disney, Adobe, Snapchat, etc.)  There was the usual news
about people
moving here or there, new products coming out, new startups being formed.
Overall, the
area is pretty hot in industry.

In any event, SIGGRAPH 2018 was fun. See you next year in Los Angeles! (By
the way
SIGGRAPH 2020 will be in Washington DC!)

Best,

-Pradeep

---
Pradeep Sen
Associate Professor
UCSB MIRAGE Lab
Dept. of Electrical & Computer Engineering
University of California, Santa Barbara
Santa Barbara, CA 93106-9560
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cs.ucsb.edu/pipermail/ilab-users/attachments/20180816/a9378f65/attachment.html>