<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi everyone,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Here are just a few of the things of interest in SIGGRAPH Days 2 and 3:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><b>PAPER TALKS</b></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><b>Computational Image Processing</b></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">H</font>u et al. "Exposure: A White-Box Photo Post-Processing Framework"</div><div class="gmail_default">This paper attempts to learn the photo post-processing pipeline. They can then automatically</div><div class="gmail_default">retouch pictures to make them look good. It's interesting because I've been talking about </div><div class="gmail_default">something related, albeit in a different way for another application.</div><div class="gmail_default"><br>He et al. "Deep Examplar-Based Colorization"<br>Another paper along the lines of training for colorizing grayscale images, but one thing that's </div><div class="gmail_default">cool is that they use an example image as a prior to reconstruct the final image's color </div><div class="gmail_default">scheme. Results are pretty impressive!</div><div class="gmail_default"><br>Zhou et al. "Non-Stationary Texture Synthesis by Adversarial Expansion"<br>First texture synthesis work I have seen that handles spatially-varying textures well without </div><div class="gmail_default">the need of user guidance. The cons is that a separate network is needed for each texture </div><div class="gmail_default">learned. I think that this overall area has more room to explore, especially for real-time </div><div class="gmail_default">applications.</div><div class="gmail_default"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Zhou et al. "</div>Stereo Magnification: Learning View Synthesis Using Multiplane Images<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">"</div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Extrapolates view from a narrow baseline of cameras using a data structure they </div><span style="font-family:arial,helvetica,sans-serif">call </span></div><div><span style="font-family:arial,helvetica,sans-serif">multiplane<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">images (MPI) (basically a set of alpha-blended planes at some known <div class="gmail_default" style="display:inline"></div></span><span style="font-family:arial,helvetica,sans-serif">depths) </span></div><div><span style="font-family:arial,helvetica,sans-serif">and <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"></div></span><span style="font-family:arial,helvetica,sans-serif">they train it<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">using deep learning. The multiplane image structure is<div class="gmail_default" style="display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">differentiable so they </span></div><div><span style="font-family:arial,helvetica,sans-serif">can train the system<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">end-to-end. Since they don't have labeled <div class="gmail_default" style="display:inline"></div></span><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">samples </div><span style="font-family:arial,helvetica,sans-serif">for real-world scenes<div class="gmail_default" style="display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">to directly train the MPI, </span><span style="font-family:arial,helvetica,sans-serif">they train them so they can<div class="gmail_default" style="display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">compute the real results at </span></div><div><span style="font-family:arial,helvetica,sans-serif">different </span><span style="font-family:arial,helvetica,sans-serif">positions with known camera </span><span style="font-family:arial,helvetica,sans-serif">positions.  Here's<div class="gmail_default" style="display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">the part I thought was most clever: </span></div><div><span style="font-family:arial,helvetica,sans-serif">to </span><span style="font-family:arial,helvetica,sans-serif">get </span><span style="font-family:arial,helvetica,sans-serif">their </span><span style="font-family:arial,helvetica,sans-serif">dataset (which is always a pain </span><span style="font-family:arial,helvetica,sans-serif">f</span><span style="font-family:arial,helvetica,sans-serif">or<div class="gmail_default" style="display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif"><div class="gmail_default" style="display:inline">these kinds of things), they </div></span><span style="font-family:arial,helvetica,sans-serif">used 7,000 </span></div><div><span style="font-family:arial,helvetica,sans-serif"><div class="gmail_default" style="display:inline">real-estate </div></span><span style="font-family:arial,helvetica,sans-serif"><div class="gmail_default" style="display:inline">videos </div>from YouTube <div class="gmail_default" style="display:inline"></div></span><span style="font-family:arial,helvetica,sans-serif">which </span><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">have a camera </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">slowly panning over </div><span style="font-family:arial,helvetica,sans-serif">scenes of </span></div><div><span style="font-family:arial,helvetica,sans-serif">houses, etc.</span><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> Very </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">good idea to get </div><span style="font-family:arial,helvetica,sans-serif">lots of data!</span></div><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">He et al. "</div>Gigapixel Panorama Video Loops<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">"</div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Very cool paper which stitches together a bunch of different videos and turns them into a </div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">navigatable </div><span style="font-family:arial,helvetica,sans-serif">video panorama.  This idea could be applied, e.g., to VR. Has anyone done<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span></div><div><span style="font-family:arial,helvetica,sans-serif">even </span><span style="font-family:arial,helvetica,sans-serif">the standard video<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div></span><span style="font-family:arial,helvetica,sans-serif">panoramas for VR?  If not, this could be an interesting area to </span></div><div><span style="font-family:arial,helvetica,sans-serif">explore as a collaboration between <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"></div></span><span style="font-family:arial,helvetica,sans-serif">MIRAGE Lab and 4Eyes lab?</span></div><div><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Aberman et al. </div>"Neural Best-Buddies: Sparse Cross-Domain Correspondence"<div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">This paper uses a hierarchy of features from a pre-trained deep CNN (VGG) to find sparse </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">correspondences across images from vary different categories. Very useful idea that could</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">potentially have many applications.</div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"></div><br></div><div><div class="gmail_default"><br></div><div class="gmail_default"><br></div><div class="gmail_default"><b>Rendering</b></div><div class="gmail_default">Anyone working on rendering or using rendering in their applications (e.g., for AR relighting) </div><div class="gmail_default">should probably check out these two new papers which talks about how to efficiently </div><div class="gmail_default">compute the integrals of spherical harmonics for lightling:</div><div class="gmail_default"><br></div>Belcour et al. "Integrating Clipped Spherical Harmonics Expansions"<br>Wang et al. "Analytic Spherical Harmonic Coefficients for Polygonal Area Lights"<br><div class="gmail_default"><br>Marco et al. "Second-Order Occlusion-Aware Volumetric Radiance Caching"<br>This paper proposes a way to compute participating media derivatives that account for<br>occlusion cha<span style="font-family:arial,helvetica,sans-serif">nges for interpolation.  They can produce some very nice results for </span></div><div class="gmail_default"><span style="font-family:arial,helvetica,sans-serif">participating media in complex scenes.</span></div></div><div class="gmail_default"><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Gruson et al. "</div>Gradient-domain Volumetric Photon Density Estimation<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">"</div><br></div><div><font face="arial, helvetica, sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Voumetric rendering in the gradient domain with path tracing, and they can apply it to a </div></font></div><div><span style="font-family:arial,helvetica,sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">variety to different methods. </div>Maybe I am biased (no pun intended), but I love these gradient-</span></div><div><span style="font-family:arial,helvetica,sans-serif">based </span><span style="font-family:arial,helvetica,sans-serif">methods.</span></div><div><font face="arial, helvetica, sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><br class="gmail-Apple-interchange-newline"></div></font></div><div><br><b>AR/VR</b><br><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Langbehn et al. "</div>In the Blink of an Eye: Leveraging Blink-Induced Suppression for </div><div>Imperceptible<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div>Position and Orientation Redirection in Virtual Reality<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">"</div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Clever idea: slightly move the scene when the viewer is blinking so that you can </div><span style="font-family:arial,helvetica,sans-serif">re-orient </span></div><div><span style="font-family:arial,helvetica,sans-serif">them </span><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"></div><span style="font-family:arial,helvetica,sans-serif">so that they can navigate a larger </span><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">virtual space in a limited physical location. Seems to </div></div><div><span style="font-family:arial,helvetica,sans-serif">work well!</span><br></div><div><span style="font-family:arial,helvetica,sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><br></div></span></div><div>Sun et al. "Towards Virtual Reality Infinite Walking: Dynamic Saccadic Redirection"<br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Kind of like the previous paper, except that the exploit our eyes' saccades to do this. These</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">are all ideas we should pursue.</div></div><div><br></div><div><div class="gmail_default"><br></div><div class="gmail_default"><b>Geometry/Modeling</b></div><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Atzmon et al. "</div>Point Convolutional Neural Networks by Extension Operators"<div class="gmail_default">One of the challenges with creating CNNs for point-cloud geometry is that it is difficult to </div><div class="gmail_default">map convolutions to point-clouds because they are unordered, not on a grid, etc. This paper</div><div class="gmail_default">comes up with the nice idea to come up with a continuous function of the surface from the</div><div class="gmail_default">point cloud and then apply a continuous convolution operator to this function. Finally, the </div><div class="gmail_default">resutling function can be sampled at the end to give the final result. This idea could be very<br>useful, e.g., for reconstructing 3D environments from structure-from-motion data.<br><br>Zsolnai-Fehér  et al. "Gaussian Material Synthesis"<br></div><div class="gmail_default">Good approach, great results! I think there is a great future in working in automatic learning</div><div class="gmail_default">for scene content creation, and material synthesis is a very important part of this.</div><div class="gmail_default"><br></div><div class="gmail_default"><br></div><div class="gmail_default">

<div class="gmail_default" style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">There were lots of other cool papers, but these were just some of the ones that stood out to me in Days 2 and 3. Let me know if anyone wants to discuss any of these or to brainstorm </div><div class="gmail_default" style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">potential extensions of this work.</div><div class="gmail_default" style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div class="gmail_default" style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div></div><div class="gmail_default"><b>SIGGRAPH KEYNOTE</b></div><div class="gmail_default"><br></div><div class="gmail_default">Rob Bredow (Head of ILM) gave the keynote on Monday and talked about strategies for</div><div class="gmail_default">success and also gave some background on the VFX on the new "Solo," where he was VFX</div><div class="gmail_default">supervisor. It's interesting to hear how many of the effects were done "old-school" with </div><div class="gmail_default">physical screens on set.</div><div class="gmail_default"><br></div><div class="gmail_default"><br></div><div class="gmail_default"><b>COMPANY ANNOUNCEMENTS</b><br><br></div><div class="gmail_default">NVIDIA had a big show on Monday afternoon where CEO Jensen Huang announced their </div><div class="gmail_default">new "Turing" class architecture Quadro RTX which can do real-time ray-tracing.  This is very </div><div class="gmail_default">exciting for those who work in photorealistic rendering or need it for their applications (e.g., </div><div class="gmail_default">AR/VR, etc.)</div><div class="gmail_default"><br></div><div class="gmail_default">I'll send out more info later.</div><div class="gmail_default"><br></div><div class="gmail_default">Best,</div><div class="gmail_default"><br></div><div class="gmail_default">-Pradeep</div><div class="gmail_default"><br></div><div class="gmail_default"><br><br></div><div class="gmail_default"><br></div><div class="gmail_default"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div><div class="m_-8475694113718620043m_6038750825782710581gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">---<div>Pradeep Sen</div><div>Associate Professor</div><div>UCSB MIRAGE Lab</div><div>Dept. of Electrical & Computer Engineering</div><div>University of California, Santa Barbara</div><div>Santa Barbara, CA <span style="background-color:rgba(255,255,255,0)">93106-9560</span></div></div></div></div>
</div></div></div>