This is it; a year has passed since WebScIT has started flying. Let take a look in the mirror on 2024 achievements.
Datalayer framework
Datalayer aims to ease the usage of powerful GPU in the cloud for your data analysis and IA training from your well know editors like JupyterLab or VS Code.
Datalayer was the first to trust us. And during 2024, we have worked hard to develop all parts of the framework from micro-services to the front-end without forgetting about the integration in JupyterLab. That involvement culminates with the opening of a private beta for the Datalayer platform.
But the opening of the private beta is just the beginning. We are continuing to support Datalayer in 2025. And we encourage you to follow them on social networks.
Mito - AI for data
We helped the awesome Saga team to improve the integration of their AI assistant into JupyterLab mito extension.
We also helped them prototyping the integration of SQL cells in JupyterLab. This is quite challenging and further polishing is required before shipping but the code basis is there.
JupyterLab
Google Summer of Code
This summer, we mentored 3 students to become new contributors to the Jupyter ecosystem thanks to the Google Summer of Code program.
Make Plugin System Data-Based
The project aimed at modernizing the JupyterLab plugin architecture. It was led by student Ashok Kumar. The initiative focused on transitioning JupyterLab’s plugin system from static imports to dynamic, on-demand loading. This shift promises to enhance performance and flexibility, ensuring that only the necessary JavaScript assets are loaded when users need them.
The goal was clear: move away from the traditional approach of importing all plugin assets upfront. Instead, Ashok sought to implement a data-driven plugin system, where extensions and their dependencies are loaded dynamically as users interact with the interface.
Key Milestones:
- Asynchronous Plugin Loading: A new Plugin Registry was introduced to manage the asynchronous loading of plugins, ensuring smoother and more efficient performance.
- Interface Upgrades: The activate function was redefined to return a promise, allowing plugins to signal when their assets are fully loaded and ready for use.
Entry Points Integration: The project successfully implemented support for several critical entry points, including:
- Document Widget factories
- Command Widget factories
- Extension Panels
- Extension Enhancements: Popular extensions like csvviewer-extension and extensionmanager-extension were updated to include entrypoint data in their package.json files, paving the way for dynamic loading.
While the project has made impressive strides, there’s still work to be done to fully realize its vision:
- Structural Refinements: standardizing the structure of entrypoints in package.json and defining clear requirements for plugin data is not finished.
- Dynamic MIME Type Support: Extensions like the json-extension need to be adapted for dynamic loading, ensuring all file types benefit from the new system.
- UI Polish: Some visual quirks remain.
All developments from this project are available in the poc/plugin-data-based branch, with several pull requests already submitted to integrate these changes upstream:
- Enable entrypoints in jupyterlab_server
- Enable Document Widget entrypoint
- Enable Command entrypoint
- Enable Extension entrypoint
Improve User Experience Using Jupyter Toolkit
Mehak Jain focused on improving JupyterLab’s user interface by integrating the Jupyter UI Toolkit into its core components. The project aimed to enhance consistency and reduce maintenance efforts across the Jupyter ecosystem.
By the end of the program, five tasks were completed, with three pull requests successfully merged into JupyterLab and the Jupyter UI Toolkit. The work included integrating unified search, button, and tree view components, as well as adding customizable widget components to the settings editor. A new icon component was also developed and implemented in JupyterLab, contributing to a more cohesive user experience.
Notable contributions included pull requests such as jupyterlab#16046, jupyterlab#16488, and jupyter-ui-toolkit#123, which reflected the project’s progress and impact.
Swiss Army Knife Builder CLI
Ronan Coutinho made significant strides in streamlining tooling for JupyterLab developers. The project aimed to extract and modularize essential build tools into a standalone package, improving maintainability for both core and extension developers while ensuring reusability across other applications.
The project successfully delivered a robust command-line interface supporting key processes: develop, build, and watch. Comprehensive unit tests were implemented using Pytest to guarantee cross-platform efficiency, and external dependencies were minimized by integrating critical tools like jlpm and jupyterlab.semver directly into the package. And the package was pre-released on PyPI, making it accessible to the broader community.
Several pull requests were merged, each contributing to the extraction and refinement of core features, such as installing JavaScript assets in development mode, automating builds, and enabling real-time asset regeneration through the watch feature.
While progress was substantial, some tasks remained ongoing, such as fully integrating the @jupyterlab/builder into the new package and updating JupyterLab’s core to adopt the tool.
Backend execution
One of the pitfall of JupyterLab and Notebook are the inability to recover results from an on-going execution if they got disconnected. To fix this, we prototyped a Jupyter extension that handle the execution request and the update of the notebook document in the server. The difference between an execution without the extension (on the left) and with the extension (on the right) can be seen in the image below:
When reloading the application, the default application does not recover the running state of the cell as it does with the extension.
Opentelemetry
When running multiple services as part of a complex application, measuring services performance and logging application behavior are critical for bug fixing and system administration. But correlating those information is not trivial and this is where the protocols and tools provided by Opentelemetry become handy. In particular it introduces the concept of trace id to identify each request going through all the services that should be attached to logs. But then comes the issue with metrics as for performance reason, the metrics are usually an aggregation of singular events rather than all the events; for example metrics will measure the median time for a request. So how can we linked the metrics to a trace id to debug for example a strangely long request. The solution proposed by Opentelemetry is the concept of exemplars that are specific metric event sampled from all events. And as they are specific event a trace id can be attached to them to correlate a metric to a trace and therefore to logs.
For example in the screenshot below, the graph on the left shows the 99% quantile of requests duration as line chart. And in the same graph, isolated data points display exemplars sampled from all the requests. Then if you pick one of those exemplar, you can access the trace id that give you access to the trace within your infrastructure.
The Python SDK was unfortunately not supporting the exemplars feature. So we pushed a PR that got merged for everybody to enjoy better telemetry with Python services.
Looking Forward
2024 was a year of meaningful collaborations and innovations, with major contributions to open-source projects like JupyterLab, Mito, and OpenTelemetry. In 2025, we hope to push these tools further, making them more powerful and accessible.
If you are interested by our work, don't hesitate to reach out.