As many of you noticed, the Zenodo upgrade last Friday didn’t go as smoothly as hoped. Afterwards Zenodo was sluggish and file uploads were painfully slow, some features didn’t quite work as expected. We’ve been working flat-out behind the scenes to steer Zenodo out of these stormy waters. We’d like to keep you updated on the issues we had, what we solved so far, and what we are still working on.
The upgrade involved orchestration of three components; Zenodo’s refactored software platform, the data, and the technical infrastructure:
The main issue after the upgrade was immediately evident that file upload/download was painfully slow (1GB file taking 1 hour instead of 20 seconds). We worked the weekend to finally discover that circumventing the front-facing load balancer alleviated the problem, and on removing it we restored Zenodo’s expected data performance (the root cause is still being investigated by the infrastructure experts).
We also had indexing issues with records which meant they didn't show up despite the data having been migrated, and despite having had successful test migration runs. We have almost finished to resolve this issue and expect the main issues to have been solved by Monday.
We want to thank you for all the incredible support and understanding that we have been receiving even under such difficulties. We’ve not ignored the functionality hiccups you’ve been reporting, we already solved some in parallel to the performance debugging. Please rest assured that we continue to work relentlessly to address all your support requests and feedback, and to address all outstanding issues in a timely manner to help you publish/access your content.
We apologise for all the inconvenience it has caused. We hope once we're over the hiccups the new Zenodo will serve you and your communities well!
Final lesson learnt: Friday 13th[**] might not have been a good day for a major release after all.
(*)
(**) We chose a Friday as Zenodo has considerable less traffic during weekends and thus as risk mitigation measure we would impact less users if there were problems (as there unfortunately turned out to be).
CERN, OpenAIRE, and the InvenioRDM open source community are excited to announce that Zenodo has moved onto our next generation underlying technical platform, InvenioRDM!
Over the past year, we've been working intensely on preparing to move Zenodo on top of a refreshed underlying technical platform, InvenioRDM. Zenodo's simple user experience and high scalability stay the same, but the underlying engine has been substantially upgraded. In addition InvenioRDM comes with a suite of new features and improvements that have been high on many of our users' wishlist.
We've significantly expanded Zenodo's collaborative features in many different areas:
You can find a comprehensive overview of new and changed behaviors in our documentation:
Checkout the user guide for a full overview on how to use the many new features:
Don't hesitate to reach out to us on support in case you find any issues on the refreshed Zenodo.
The engine behind the new Zenodo, InvenioRDM, is available for institutions worldwide and can be customized to your institutional or domain-specific needs, staying close to Zenodo's simple and seamless user experience.
We're proud to have built InvenioRDM together with 27 other partner organisations across the globe as an open source community around the simple vision of providing a great user experience to researchers with the ability to handle large amounts of data. Next, we're working towards making InvenioRDM a fully collaborative platform that empowers its users to share and preserve research.
The new Zenodo platform was made possible through a strong open collaboration.
We're grateful to our project funders who have enabled us to build and operate Zenodo for the long tail of science over the past 10 years:
We're grateful to all the InvenioRDM partners who shared our vision and helped build an amazing next-generation platform. Thanks to:
We'd like to extend a big thank you to our users and project partners for testing, feedback, and patience, in particular:
The CERN team who worked on the implementation and migration:
Including many other highly experienced CERN colleagues from the IT Department who helped with all things infrastructure (admin, compute, databases, network, openshift, outreach, search, storage).
The InvenioRDM open source community who contributed with design, development, testing, requirements, outreach, documentation, and expertise:
See the original announcement.
We unfortunately have to postpone the planned migration on September 29th 6-8 UTC. The new migration date is October 13 from 06:00-08:00 UTC. We apologise for any inconvenience it may cause.
During a final trial migration run last week we have discovered some issues that we've been trying to address in time to be ready to start the final migration process today. Unfortunately, we've not be able to fully address the issues and as a precautionary measure we're postponing the migration.
Zenodo has over the past 10 years grown in both scale and size, and thus migrating the full system to a new technical platform is a large undertaking. In order to minimize service disruption for our users, we're taking an incremental approach. This approach works by migrating a snapshot of the existing system, and afterwards incrementally apply changes in the existing system to the new system. This method enables us to keep the downtime of Zenodo to about 1-2 hours instead of the ~3 days which full migration takes due to Zenodo's scale and size. The incremental approach however is also more technically challenging, and it is through our extensive quality assurance checks that we discovered issues that we need further time to investigate.
If you're eager to learn more abou the new platform, then you can find a comprehensive overview over what's new and what's changed on the following links: