ICLR 2020 Online: a first review
As Zeta Alpha, we joined the ICLR conference with (almost) the whole team, illustrating one of the upsides of going online. A team of a dozen people could participate in the online version of a conference for the budget that would otherwise easily be spent for a single individual traveling to a physical event. A great opportunity not only for smaller companies, but probably even more so for students and members of under-funded institutions! Here is a personal review on the ICLR 2020 conference, along with some quantitative data analysis.
Fully Online, a First
The ICLR conference in its 2020 edition was the first edition to take place fully online. It was not the very first Machine Learning-related conference that was forced to that move, but still a pioneer in this area. The organizers did an amazing job to build a unique online conference platform in record time with video presentations of posters and keynotes for asynchronous access, visualization, browsing and search in all accepted papers, video chats with authors on Zoom, and a group chat platform on Rocket Chat. So how did this work out for the participants? As good as the real thing?
After all, conferences are not only about presentations of state-of-the-art papers, but also an occasion for socialising in the community. You can meet the people behind these amazing new works, ask them to clarify things you would like to understand in more depth, and have fun together. How would that work online? Whereas mingling seems much harder in an online event, it might actually even be easier for some.
Approaching others in person can be challenging at a large conference. People are busy, and whether or not you manage to find and actively take your chance to discuss with that one author whose brain you wanted to pick, often depends on whether you both happen to coincide. If not, that is bad luck, and a missed opportunity for both.
As data enthusiasts at Zeta Alpha, we have found the text-based part of the online communication beneficial in another way: using the public chat rooms to perform some quantitative sociological analyses on the conference promised fun for nerds, but also some actually interesting insights!
Before ICLR 2020 started, the largest ever in terms of participants and accepted papers, we used our platform for finding interesting papers. We identified already famous and influential papers up-front, and used insights coming from our semantic search engine to approximate relevance of papers from different angles.
Now that the conference is over, its chat system has provided another perspective with which we are happy to get our hands dirty for more insights.
For starters, each conference poster had a dedicated chat room in which the authors could be found. Definitely during defined time slots, but often also outside of these. Like in real life, discussions sometimes went on even after the authors had left. In order to see which posters were most actively discussed, we looked at the activity in each of these rooms. To give the data scientist in us a particular pleasure, the distribution follows a nice Zipf distribution pattern.
Here are the twenty most actively discussed papers in the ICLR chat fora:
Drawing Early-Bird Tickets: Toward More Efficient Training of Deep Networks by Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, Yingyan Lin
Behaviour Suite for Reinforcement Learning by Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, Hado Van Hasselt
Interestingly, the most discussed paper by Khandelwal et al. was not in our list of 20 most-cited papers before, and it was not the only new contribution to attract a lot of attention. ALBERT, on the other hand, a much cited contender in the lightweight class of the BERT model league, apparently did not require as much discussion as some others during the conference.
Not having done any qualitative analysis, we leave further conclusions up to the reader. Feel free to share your own thoughts with us!
Similar to the poster sessions, the social events also had their own chat rooms. Discussions took place during the whole conference, and we looked at which communities were most active. The 35 “social” channels were formed around various dimensions such as specific subfields, geographics, as well as social, ethical, and political topics. The most active ones were:
Topics in Language Research
Open source tools and practices in state-of-the-art DL research
The RL Social
The Bitter Lesson for AI
This list shows that NLP is, maybe not surprisingly given the surge of attention to this field, the main topic at the ICLR. Open Source Software development is the norm in both academic and industrial research. Furthermore, we see that topics around diversity, most prominently represented by the “BlackInAI” channel, and meta-topics (“The Bitter Lesson for AI”) have been discussed very actively. Reinforcement learning remains a hot topic, illustrated not only by the amount of conference papers, but also by the chat activity.
Online: Pros and Cons
Summing up, and looking back at the first virtual ICLR conference as a whole, here is our summary of the advantages and disadvantages.
Lower hurdle for participation (financially, visa-related)
No traveling: less environmentally harmful and less individual stress
(Mostly) asynchronous: little individual adaptation to schedule required
Amazing technical platform; thanks for the great organization! This alone is a great addition to the regular conference experience.
Still no major Machine Learning/NLP conference in Africa
Harder to socialize, at least for some (see above)
Harder to fully focus on conference when being physically at your normal workplace
I’m curious what other people think, so feel free to discuss with us on Twitter.
Also, if you have participated in ICLR 2020, don’t forget to take the survey. This will help the organizers to learn and improve for next year's ICLR conference. While it’s unclear if that can be a physical event again, and whether or not we would even want it to be, at least the programme chairs are already known.
We are totally excited for next years ICLR! Hopefully a good mix of in person meeting and the best of this year's awesome online experience.