<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi everyone,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>Here is my last update from Vancouver, this one for Days 4 and 5. Enjoy!</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><b>PAPER TALKS</b></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><b>Rendering</b></div><br>Vevoda et al. "Bayesian online regression for adaptive direct illumination sampling"<div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">This paper performs online learning of direct lighting to do importance sampling of light </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">sources for path tracing techniques. This is similar to what we were exploring when we </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">(Steve and I) first started the project with Pixar last summer.  Should we have just written </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">that up and published it? Probably! It is good to observe when one becomes so focused on </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">the "deep" problems that we miss the lower hanging fruit. We need to read through this and </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">figure out what is still salvagable of that original work.</div><div><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><b>AR/VR</b></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Li et al. "</div>Scene-Aware Audio for 360° Videos<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">"</div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I didn't go to this talk, but it looks interesting. The basic idea is that standard audio does not </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">work well for 360 videos because it doesn't account with how people turn their heads as they</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">are looking around. I.e., in traditional approaches, the sound always follows the user, so that</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">as you rotate your head the sound follows.  In this approach, they code the sound in such a</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">way so that it feels like it is located at that same physical location regardless of how you turn</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">your head. I don't know much about sound, but it seems interesting.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"></div><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><b>3D Reconstruction</b></div><div class="gmail_default"><br>Hedman et al. "Instant 3D Photography"<br><font face="arial, helvetica, sans-serif">Aligns multiple RGBD images and stitches them together to turn them into a mesh so that </font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">they </font><font face="arial, helvetica, sans-serif">can </font><span style="font-family:arial,helvetica,sans-serif">be viewed in VR. Very impressive results!</span></div><div class="gmail_default"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Whelan et al. "</div>Reconstructing Scenes with Mirror and Glass Surfaces<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">"</div><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">This paper addresses the problem of scanning glass/mirror surfaces for 3D scanning. They </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">detect mirrors by the observation that if you can see yourself (i.e., the camera) then you are </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">likely looking in a mirror.  So they put an "April Tag" on the camera that can be seen in the </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">images and they use it to automatically recognize the mirrors and remove them, to produce </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">nice results. (This was the paper that Chris Sweeney told us about.)</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Wu et al. "</div>Full 3D Reconstruction of Transparent Objects<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">"</div><br>Proposes a way to 3D scan transparent objects<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> by finding </div><span style="font-family:arial,helvetica,sans-serif">ray-ray correspondences entering </span></div><div><span style="font-family:arial,helvetica,sans-serif">the<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">transp</span>arent object<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div>from structured illumination and then optimizing the surface.</div><div><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Lindell et al. </div>"Single-Photon 3D Imaging with Deep Sensor Fusion"</div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">This method proposes a way to combine information from a low-resolution low-sample count</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">time-of-flight imager with a high-res intensity image in order to produce high fidelity depth maps.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div>Yin et al. "P2P-NET: Bidirectional Point Displacement Net for Shape Transform"</div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">This paper proposes to a deep learning network that moves points around to form structures.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">They do this with a bidirectional training, kind of like Cycle GAN.  I think there could be</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">applications of these ideas to rendering,</div><br></div><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><b>Relighting</b></div><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Xu et al. "</div>Deep Image-Based Relighting from Optimal Sparse Samples<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">"</div></div><div><font face="arial, helvetica, sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">This paper tries to solve the image-based relighting problem using deep learning. </div></font></div><div><font face="arial, helvetica, sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">While </div></font><span style="font-family:arial,helvetica,sans-serif">normally<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">light transport is measured independently from scene to scene (essentially </span></div><div><span style="font-family:arial,helvetica,sans-serif">almost </span><span style="font-family:arial,helvetica,sans-serif">overfitting<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">to <div class="gmail_default" style="display:inline"></div></span><span style="font-family:arial,helvetica,sans-serif">each individual scene), they trained a system that would take five </span></div><div><span style="font-family:arial,helvetica,sans-serif">captured </span><span style="font-family:arial,helvetica,sans-serif">images and then<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">learn<div class="gmail_default" style="display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">to interpolate to match a lot of lighting configurations. </span></div><div><span style="font-family:arial,helvetica,sans-serif">Interesting that </span><span style="font-family:arial,helvetica,sans-serif">they generated<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">random objects<div class="gmail_default" style="display:inline"> </div><div class="gmail_default" style="display:inline"></div><div class="gmail_default" style="display:inline"></div></span><span style="font-family:arial,helvetica,sans-serif"><div class="gmail_default" style="display:inline">f</div>or their training<div class="gmail_default" style="display:inline"> procedure simply by </div></span></div><div><span style="font-family:arial,helvetica,sans-serif">deforming meshes randomly.</span><br></div><div><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><b>Computational Photography</b></div><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Sitzmann et al. "</div>End-to-end Optimization of Optics and Image Processing for Achromatic </div><div>Extended Depth of Field and Super-resolution Imaging<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">"</div></div><div>Very nice id<span style="font-family:arial,helvetica,sans-serif">ea which started with a question: is the relationship between the camera optics </span></div><div><span style="font-family:arial,helvetica,sans-serif">and<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">the point spread function (PSF) differentiable? If so, we can put the optical design within </span></div><div><span style="font-family:arial,helvetica,sans-serif">an<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">optimization to figure out how to construct a lens that would achieve a particular effect.  </span></div><div><span style="font-family:arial,helvetica,sans-serif">For<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">example, they solve for the optical height map that would give them a sharp image, and </span></div><div><span style="font-family:arial,helvetica,sans-serif">they<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">found something similar to a Fresnel lens. They did end to end training along with the </span></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">reconstruction algorithm, and were able to apply it to a couple of real problems, such as</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">addressing all-in-focus imaging, etc.  Awesome!<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Deschaintre et al. "</font>Single-Image SVBRDF Capture with a Rendering-Aware Deep Network"<br><font face="arial, helvetica, sans-serif">This paper proposes to use a CNN that takes as input a single image and outputs a spatially</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">varying BRDF. Specifically, their BRDF model consists of the </font><span style="font-family:arial,helvetica,sans-serif">surface normals, as well as</span></div><div class="gmail_default"><span style="font-family:arial,helvetica,sans-serif">spatially-varying diffuse, roughness, and specular </span><span style="font-family:arial,helvetica,sans-serif">terms.. One key is that rather than </span></div><div class="gmail_default"><span style="font-family:arial,helvetica,sans-serif">compare against the GT maps for training, they do their loss against their rendering, which</span></div><div class="gmail_default"><span style="font-family:arial,helvetica,sans-serif">requires a differentiable rendering system. </span><span style="font-family:arial,helvetica,sans-serif">Abhishek, you may want to look at </span><span style="font-family:arial,helvetica,sans-serif">this paper as it</span></div><div class="gmail_default"><span style="font-family:arial,helvetica,sans-serif">is related to what you are working on and we have discussed many similar things.</span></div><div class="gmail_default"><span style="font-family:arial,helvetica,sans-serif"><br></span></div><div class="gmail_default"><span style="font-family:arial,helvetica,sans-serif"><br></span></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><b>HPG</b></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Poms et al. </div>"Scanner: Efficient Video Analysis at Scale"<br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">This paper proposes a</div> software system for large-scale video <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">processing that allows the user </div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">to </div><span style="font-family:arial,helvetica,sans-serif">run image processing tasks on large sets of videos on heterogeneous computing clusters.</span></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><b>COURSES<br></b><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">There was a pretty interesting course on Rendering and Machine Learning on Thursday 2-</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">5pm. The course notes seem to have a lot of links to relevant literature, so we should pore </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">through that carefully.  For example, Jan Novak talked about some new work they just </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">released to arXiv that seems really related to the project Steve was working on for the past </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">couple of years:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><a href="https://arxiv.org/pdf/1808.03856.pdf" target="_blank">https://arxiv.org/pdf/1808.<wbr>03856.pdf</a></font><br></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">They are using machine learning to learn an importance sampling function. We should read</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">through it and discuss.</font></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><b>OTHER<br></b><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">As usual, there were also lots parties (Facebook had a party for the first time, along with the <br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">usuals NVIDIA, Disney, Adobe, Snapchat, etc.)  There was the usual news about people</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">moving here or there, new products coming out, new startups being formed. Overall, the </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">area is pretty hot in industry.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">In any event, SIGGRAPH 2018 was fun. See you next year in Los Angeles! (By the way </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">SIGGRAPH 2020 will be in Washington DC!)</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Best,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">-Pradeep</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div><div class="m_267812134236013513gmail-m_-4833616534398085534m_-5456758274088228549m_4619770503675453779m_5747540326335282598m_-8496775885073271083gmail_signature"><div dir="ltr">---<div>Pradeep Sen</div><div>Associate Professor</div><div>UCSB MIRAGE Lab</div><div>Dept. of Electrical & Computer Engineering</div><div>University of California, Santa Barbara</div><div>Santa Barbara, CA <span style="background-color:rgba(255,255,255,0)">93106-9560</span></div></div></div></div>
</div></div></div></div>