A Guide to NeurIPS 2023 — 7 Research Areas and 10 Spotlight Papers to Read

NeurIPS is back in New Orleans, with record-breaking numbers across the board: 3584 main papers, 58 workshops, 14 tutorials, and 8 keynote talks. Featuring new developments in fields such as Language Modeling, Reinforcement Learning, Machine Learning Optimization, Representation Learning, and Diffusion Models, this year's edition is once again packed with world-class AI research. At Zeta Alpha, we have curated a short guide to help you navigate the conference, highlighting some of the papers and areas that grabbed our attention.

Image by Zeta Alpha.

Before we start: we will be covering these highlights in this month's installment of our Trends in AI webinar, so join us live this Thursday, December 14th, either in person at LAB42 or online on Zoom, to discuss recent developments in the field, along with a highlight of our favorites from NeurIPS and other papers from the past month!

To visualize the content that will be presented at the conference, we created a semantic map of the published papers, categorizing them into clusters based on their similarity using the VOSviewer tool and a bit of our secret sauce using Language Models to automatically label the cluster centroids.

Pro tip: view the graph below in full screen (or in a new tab) to navigate it freely, and discover the papers of interest in each cluster at your own pace!

As this high-level overview can still be overwhelming, we dove deeper into each research area presented in the visualization above and selected some of the spotlight papers that are worth reading in more detail. We have organized our top-10 spotlight papers in a Zeta Alpha tag for your browsing convenience, and you can also find all 77 oral papers here.

You will notice that most of these papers are already widely known, with some having already received dozens of citations and inspired follow-up work, as they have been public on arXiv for a few months now. Despite this, this overview also serves as a round-up of some of the most influential works in 2023, with their publication at this prestigious venue being yet another proof of their impact.

Throughout this guide, you can use Zeta Alpha to browse more of the papers that will be presented at NeurIPS this year, by clicking on the "🔎 Find Similar" button next to each of the featured titles, which will take you to our platform to discover similar work using neural search.

Before we dive in, we wanted to give a shoutout to the excellent work that came out of Amsterdam, which is where our home base is located. You can explore the contributions from researchers affiliated with the University of Amsterdam to this year's conference directly in Zeta Alpha!

---

⚠️ Disclaimer ⚠️ Of course, this cannot be a fully comprehensive guide, given the sheer number of papers we're working with here, but we hope this is a useful entry point to the conference.

1. Large Language Models

Spotlight papers:

1️⃣ QLoRA: Efficient Finetuning of Quantized LLMs — 🔎 Find Similar

💡 Researchers from the University of Washington have developed QLoRA, an efficient finetuning approach for quantized language models, achieving state-of-the-art results on the Vicuna benchmark and reducing memory usage while maintaining performance.

2️⃣ Tree of Thoughts: Deliberate Problem Solving with Large Language Models — 🔎 Find Similar

💡 This paper presents the "Tree of Thoughts" (ToT) framework, which enhances language model inference by introducing deliberate problem-solving through the generation and evaluation of coherent text units, demonstrating improved problem-solving on tasks requiring planning or search.

3️⃣ Are Emergent Abilities of Large Language Models a Mirage? — 🔎 Find Similar

💡 This work questions emergent abilities in large language models, asserting that these skills are shaped by metric choices, not inherent model traits. By altering evaluation metrics, the authors demonstrate how observed abilities can vanish or emerge, emphasizing the crucial role of metric selection and the necessity for rigorous controls in assessing LLM capabilities.

More in this research area:

• Guiding Large Language Models via Directional Stimulus Prompting — 🔎 Find Similar

• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision — 🔎 Find Similar

• H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — 🔎 Find Similar

• ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings — 🔎 Find Similar

• ToolQA: A Dataset for LLM Question Answering with External Tools — 🔎 Find Similar

2. Reinforcement Learning

Spotlight paper:

4️⃣ Direct Preference Optimization: Your Language Model is Secretly a Reward Model — 🔎 Find Similar

💡 Direct Preference Optimization (DPO), a novel algorithm for training language models, aligns with human preferences by directly optimizing the policy through a binary cross-entropy objective, outperforming methods like PPO while being computationally efficient and simple to implement.

More in this research area:

• Coherent Soft Imitation Learning — 🔎 Find Similar

• HIQL: Offline Goal-Conditioned RL with Latent States as Actions — 🔎 Find Similar

• Self-Supervised Reinforcement Learning that Transfers using Random Features — 🔎 Find Similar

• On the Importance of Exploration for Generalization in Reinforcement Learning — 🔎 Find Similar

• Learning from Active Human Involvement through Proxy Value Propagation — 🔎 Find Similar

3. Neural Network Optimization

Spotlight papers:

5️⃣ Fine-Tuning Language Models with Just Forward Passes — 🔎 Find Similar

💡 Researchers from Princeton University have developed MeZO, a memory-efficient zeroth-order optimizer that significantly reduces memory requirements for fine-tuning large language models, achieves comparable performance to backpropagation with reduced memory usage, and is compatible with non-differentiable objectives and various tuning techniques.

6️⃣ Scaling Data-Constrained Language Models — 🔎 Find Similar

💡 This paper explores the scaling of language models in data-constrained scenarios, suggesting that training models with multiple epochs of repeated data can yield negligible changes in loss compared to using unique data. The authors propose a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters, while also discussing strategies for mitigating data scarcity, such as augmenting the training dataset with code data or removing commonly used filters.

More in this research area:

• Private (Stochastic) Non-Convex Optimization Revisited: Second-Order Stationary Points and Excess Risks — 🔎 Find Similar

• Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Models — 🔎 Find Similar

• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection — 🔎 Find Similar

• Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension — 🔎 Find Similar

• Learning in the Presence of Low-dimensional Structure: A Spiked Random Matrix Perspective — 🔎 Find Similar

4. Equivariant Representation Learning

Spotlight paper:

7️⃣ Clifford Group Equivariant Neural Networks — 🔎 Find Similar

💡 Clifford Group Equivariant Neural Networks, utilizing Clifford algebra and the Clifford group, present a novel method for constructing O(n)- and E(n)-equivariant models, achieving state-of-the-art performance in tasks like n-body simulations, convex hull estimation, and top tagging in particle physics.

More in this research area:

• Learning Linear Causal Representations from Interventions under General Nonlinear Mixing — 🔎 Find Similar

• Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules — 🔎 Find Similar

• AbDiffuser: full-atom generation of in-vitro functioning antibodies — 🔎 Find Similar

• Fair Graph Distillation — 🔎 Find Similar

• Self-Supervised Learning with Lie Symmetries for Partial Differential Equations — 🔎Find Similar

5. Adversarial Attacks & Generalization

Spotlight paper:

8️⃣ Jailbroken: How does LLM Safety Training Fail? — 🔎 Find Similar

💡 Despite safety training, large language models remain susceptible to jailbreak attacks due to competing objectives and mismatched generalization, revealing persistent vulnerabilities that scaling alone cannot eliminate, underscoring the crucial necessity for safety-capability parity and addressing limitations in current safety training methods.

More in this research area:

• Learning Sample Difficulty from Pre-trained Models for Reliable Prediction — 🔎 Find Similar

• Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection — 🔎 Find Similar

• StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners — 🔎 Find Similar

• Harnessing Hard Mixed Samples with Decoupled Regularizer — 🔎 Find Similar

• Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization — 🔎 Find Similar

6. Diffusion Models

Spotlight paper:

9️⃣ Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation — 🔎 Find Similar

💡 This paper reveals the close relationship between diffusion model objectives and the Evidence Lower Bound (ELBO), showing that diffusion objectives can be understood as weighted integrals of ELBOs and proposing new monotonic weightings that achieve state-of-the-art scores on ImageNet.

More in this research area:

• CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulation — 🔎 Find Similar

• Real-World Image Variation by Aligning Diffusion Inversion Chain — 🔎 Find Similar

• Collaborative Score Distillation for Consistent Visual Editing — 🔎 Find Similar

• UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild — 🔎 Find Similar

• Controlling Text-to-Image Diffusion by Orthogonal Finetuning — 🔎 Find Similar

7. 3D Object Representations

Spotlight paper:

🔟 Rotating Features for Object Discovery — 🔎 Find Similar

💡 This paper introduces Rotating Features, a novel approach for object discovery in machine learning, leveraging higher-dimensional generalization of complex-valued features to represent a greater number of objects concurrently, showcasing scalability across simple toy datasets to complex real-world datasets and offering a new paradigm to tackle the binding problem.

More in this research area:

• Autodecoding Latent 3D Diffusion Models — 🔎 Find Similar

• Structure from Duplicates: Neural Inverse Graphics from a Pile of Objects — 🔎 Find Similar

• PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generation — 🔎 Find Similar

• Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion — 🔎 Find Similar

• Slot-guided Volumetric Object Radiance Fields — 🔎 Find Similar

---

Did we miss anything major? Let us know what you think on X (formerly Twitter): @ZetaVector.