• 1. Leaving Stability
• 2. My roots with open science & OnNeuro
• 3. Transitioning to neuroAI
• 4. From OnNeuro to LAION
• 5. From LAION to MedARC
• 6. Departure
• 7. Conclusion

Leaving Stability

I resigned from Stability AI after over a year working as Head of NeuroAI. I’m now looking into new ways to foster fully transparent open-source science and conduct research on neuroAI and medical AI more broadly.

A personal retrospective below touches on my journey with open science communitiesI plan to write more on science-in-the-open soon, subscribe to my mailing list here https://groups.google.com/g/paulscotti and how Stability AI’s role in the “science-in-the-open” movement was such a pivotal phenomenon to have come and gone.

My roots with open science & OnNeuro

Back in 2017 I had an idea—what if I could coordinate an international network of likeminded neuroscientists and psychologists together through an online community? We could share our ideas and progress as they happened, and stay in touch through regular online journal clubs.

I had just moved to Columbus, Ohio for my PhD studies and was feeling quite isolated. Relatedly, I was thinking about how academics typically work in isolation within their lab. Usually the earliest academics share their ideas and work outside of their own lab is at infrequent academic conferences. And if they don’t share at a conference then they might not share their findings until years after the work was completed, as a finished peer-reviewed research publication. It just seems like an obviously inefficient system for science progress as a whole. I wanted to build an online community of neuroscientists where we could all learn and grow together as emerging academics.

Nowadays this idea seems ordinary; online journal clubs are commonplace and there’s plenty of internet communities dedicated to academic subfields. But remember, this is back in 2017! Before the COVID pandemic and before the boom of Discord research communities! The notion that anyone could go on their laptop and attend a live presentation and do Q&A with an esteemed professor across the globe was not at all a regular occurrence.

I tapped into my limited networks, advertised at academic conferences, and built a modest group of researchersa deep and sincere appreciation to Molly McKinney and Allison Londerée and the OSU CCBBI student group for being critical to OnNeuro gaining traction at all! who regularly attended OnNeuro meetings. I also expanded from online journal clubs to live-streaming all the neuro lectures at my university that I could, with recordings uploaded to YouTube. I tried to convince people at other institutions to do the same, but I struggled because it was a heavy undertaking it was not as simple as just having Zoom open in the background, I spent too many hours tinkering with OBS and Zoom to get live-streaming to work for various locations for them to communicate with IT, get approval from speakers for their lecture to be recorded and shared online, and get approval from university departments.

When COVID hit in 2020, Tim Vogels and Panos Bozelos started an initiative called WorldWideNeuro. They essentially had the same idea as me but they started at the perfect time to tap into the pandemic ceasing travel, and they actually had the networking power to convince academics across tons of universities to start sharing their in-person lectures and seminars online. 2020 also happened to be the year I was finishing up my Ph.D. and so I was limited in free time. This demotivated me from continuing to put effort into OnNeuro, and I ultimately gave it away to the Ohio State University CCBBI Student Group to continue recording meetings and putting the content on YouTube (which I’m happy to see they are still doing!).

Transitioning to neuroAI

I feel I owe a lot of my grad school success to open science because I believe it was talking about OnNeuro in my personal statement that helped award me an NSF GRFP fellowshipthe National Science Foundation Graduate Research Fellowship Program awards ~35K a year for 3 years to work on research without worrying about teaching obligations or other research grants for funding. This fellowship funded my PhD studies where my dissertation involved computational modeling to reconstruct visual perception (simple features like color and orientation) from fMRI brain activity. This work helped land me a postdoc, where I left Ohio State University and joined Princeton University in 2022.

My postdoc project tackled decoding of visual memory. I wanted to look into using AI image generation models to help reconstruct images from fMRI brain patterns using a new dataset called the Natural Scenes Dataset¹. This started my transition from computational neuroimaging into the newly coined neuroAI fieldbtw who first coined the term neuroAI? or did it emerge naturally as the way to refer to this field?, as I took to the internet to improve my deep learning expertise.

From OnNeuro to LAION

I discovered the LAION Discord server and figured that seemed like a good place to connect with other passionate AI researchers and improve my skillset so I could best tackle my postdoc project. But who would have thought that there would already be a channel inside of the LAION Discord specifically tackling reconstruction of visual perception from brain activity?

I introduced myself to the group and quickly thereafter became the project lead for what later became MindEye². Tanishq (aka ilovescience) welcomed me into the group and our small group quickly started making progress with weekly meetings and frequent asynchronous updates in the Discord thread. Every so often a new person would show up and join the team, where we all were simply contributing to a shared public GitHub repository and posting about our updates as we tried different ideas out. One such team member was Atmadeep (aka aka Atom_101) who contributed significantly and became co-first author with me on the MindEye paper. I learned a ton during this time—clearly much more than I would have if I were independently researching as a postdoc—our scrappy mishmash team of online volunteers shared ideas and trained large-scale multi-gpu neural networks and our own denoising diffusion models from scratch.

From LAION to MedARC

After a few months, Tanishq was hired as research director at Stability AI and he created his own Discord server named MedARC to host various medical AI research projects. Tanishq offered me the opportunity to join Stability AI as a full-time employee where my job would be to lead and organize neuroAI open-source projects. It seemed like a no-brainer—I would continue to work on the research I was most passionate about, contribute to open science, and I would get paid more. So the group moved over to MedARC and we resumed work.

In 2023, MindEye was accepted (and featured as spotlight) to NeurIPS. We achieved state-of-the-art performance reconstructing visual perception from fMRI brain patterns. It received a great deal of public attention and I was excited that this novel science-in-the-open approach seemed to be working so well. NeurIPS was also the first time I had a chance to meet some of the MedARC community in person!

We subsequently developed a follow-up, MindEye2³, which was accepted to ICML and showcased further improvements to state-of-the-art and also specifically improved performance in low-sample settings via shared subject models (as opposed to previous work which trained models independently for every participant in the dataset).

Departure

In hindsight, our situation was indeed a bit too good to be true. Stability compute supported not just MedARC’s endeavors but a whole host of other open science Discord channels (e.g., EleutherAI, LAION, OpenBioML, CarperAI). Stability AI provided free compute to volunteers working on open-source research in these communities and it enabled us to fully leverage crowd-sourced transparent science without the barriers typical to these sorts of projects. We used the Stability HPC as a shared file system and contributors could easily run different ablations and model trainings using the same compute setup. Understandly, a setup like this was not the most sustainable for a for-profit startup—these open-source science projects were not directly contributing revenue to the company—and the external HPC was eventually shut down.

Conclusion

Stability AI was my first step outside of the traditional academic pipeline and I am incredibly grateful for the opportunity to spend over a year working as the principal investigator of the MedARC Neuroimaging and AI Lab, collaborating across academia, industry, and an international group of passionate and talented research volunteers. I am convinced that the fully transparent science-in-the-open approach, if supported in the right wayNote that I don't blame Stability AI for their decision to shut down the external HPC. It was sad to see but it was perhaps the right move in terms of keeping the startup sustainable from a business perspective., still hasn’t reached its full potential for scientific breakthroughs. And now, having left Stability AI, I am also in a new phase of leaving stability in my life overall. Stability AI and academia have been stable pillars I relied on and now my career is inherently less stable—I figure I need to be more comfortable with instability if I wish to wholeheartedly continue embracing my ideals surrounding open science.

Thanks to Patrick Mineault and Tanishq Abraham for inspiring me to start writing thoughts in blog form.