Vidyasangam (विद्यासंगम)

A documentary around Scientific Python, Pyodide, and my open source journey so far

Agriya Khetarpal

About me 😁

  • Computer science + applied math background
  • Twenty-two years old and graduated in late 2024
  • Software engineer at Quansight
  • Privileged to contribute to the Scientific Python ecosystem and help maintain the Pyodide ecosystem

  • Other work: autograd, and the PyBaMM ecosystem (Python Battery Mathematical Modelling)

  • Interested in
    • Python packaging 📦🐍
    • Scientific computing ➗🧪
    • Compilers and toolchains 🛠️⛓️
    • Documentation and technical writing 📝🌉
    • ...and more 👾
  • How I got into working on open source and Scientific Python
  • The Pyodide ecosystem
    • Python, running in a WebAssembly environment
    • Pyodide at the heart of Scientific Python
    • Pyodide-powered interactive documentation
  • Three learnings from open-source
  • What is it like to work on open-source, full-time?
  • Takeaways from this talk

Outline 🎙️

  • Google Summer of Code 2023
  • PyBaMM (Python Battery Mathematical Modelling) – a NumFOCUS sponsored project

Humble beginnings 👣

Humble beginnings 👣

Learning I

The notion of open source software

defies conventional boundaries

around software in general

defies conventional boundaries

open source software

OSS defies boundaries 🚧

  • Community contributions are (mostly) welcomed
  • Several moving parts to software
  • All with equally rewarding work
    • Core libraries
    • Infrastructure around documentation
    • Technical writing
    • DevOps and CI/CD improvements, builds, packaging
    • Developer and contributor experience

Identifying gaps within open

source boundaries

Realisations and realisms 🙈

  • Adding value to ecosystems where no one else has before brings value to you
  • No one working on open-source software has the time to do everything
  • So, everyone does everything "they can"

PyBaMM then

  • No parallel jobs in CI
  • Fifty minutes for CI to run on every commit
  • Less emphasis on proper packaging standards for a compiled project, causing issues with installation
  • Faster CI
  • Nicer documentation
  • Migration to scikit-build-core as a new build backend
  • A year for me maintaining PyBaMM, and I'm still contributing to various parts of it – docs, infra, solvers, packaging, and more!

PyBaMM now

...thus, Lesson II

The world of open source
is all about                        

small wins

The Pyodide ecosystem

Python implementations 🐍

  • CPython – the most widely used (and the reference) implementation of the Python programming language
  • However; other implementations, exist, too!

IronPython

IronPython

GraalPy

RustPython

  • CPython, cross-compiled via the Emscripten toolchain to WebAssembly (WASM)
  • Entire distribution in itself, rather than just an implementation
  • Runs on all browsers
    • Chromium derivatives, Firefox, and Safari
    • Even on Android and iOS!
    • and in JavaScript runtimes, such as Node.js, Bun, Deno
    • Write once, run anywhere

WebAssembly and Emscripten

  • Compiles C/C++ code to WASM (wasm32-emscripten) via LLVM
  • Provides an entire toolchain for this, including the compiler and the linker, and build utilities
  • emcc, as a drop-in replacement for gcc
  • Binary format for a stack-based VM, intended as a compilation target
  • Inherently designed to be secure and to run in constrained environments
  • Supported by most modern browsers

Python + WASM?

  • RustPython and Pyodide both offer an implementation of the Python programming language that can run within a WASM runtime
  • However, Pyodide can run CPython extension modules – while this use case is limited for RustPython

More than just

+

Python/JavaScript

Foreign function interface

WebAssembly + JavaScript stdlib

{

{}

Glue code

micropip

(CDN)

225+ packages!

(PyPI)

pure Python wheels

and essential low-level libraries

                                 Python in WebAssembly

Use cases of Pyodide 🎓

  • Online educational content and learning materials
  • Web apps, leveraging interfaces with popular Web APIs
  • On-device machine learning and AI inference
  • ...and more!

A demo: let's play a game 🎮

A game made using pygame-ce and SDL2

Play along!

https://agriyakhetarpal.github.io/sudoku/ – (works better on laptops)

XKCD 1987

  • Documentation for scientific software needs to do more
  • Some guidelines
    • Explanatory (descriptive)
    • Illustrative (graphs, diagrams, visual aids, and more)
    • Interactive (a boost to user experience)
https://diataxis.fr/

Motivation 🙇

Diátaxis, from the Ancient Greek δῐᾰ́τᾰξῐς: dia (across) and taxis (arrangement).

A new literate programming paradigm for Python: in-browser execution via WASM

New entrant to the Jupyter ecosystem!

Pyodide's limitations

  • Some modules from Python standard library are unavailable
    • No threading or multiprocessing
    • Limited sockets support (no socket from the stdlib)
  • Browser security restrictions – here, the browser is the runtime, and a browser can't replace an operating system
  • Operating system features such as access to peripherals are restricted, and means of accessing the file system within the runtime are limited
  • Using multiple threads is limited to just web workers and service workers

Pyodide at the heart of 

Scientific 

Python

Supported Scientific Python packages

Most of them are now built against WASM in CI, and all of them run in Pyodide!

Extending support for a new platform 🚉

  • Fix compilation errors and/or failing tests
  • Fix a few crashes and fatal errors at runtime
  • Organise a CI job, so that nothing breaks in the future
  • Quite straightforward (but also complex!)
  • Implemented for seven projects by now

The structure of a Pyodide CI job 🗼

  • Builds your package out-of-tree (outside the Pyodide repository); as opposed to in-tree
  • Test suites can be run using pytest, or other test runners

Start

Check out sources

Set up
Python and Emscripten

pyodide build

Invoke

Run tests

Install Node.js

End

Pyodide wheels 🛞

{distribution}-{version}-{python tag}-{abi tag}-{platform tag}.whl

Pure Python wheels are named mypackage-0.1.0-py3-none-any.whl

Compiled wheels get tagged as

mypackage-0.1.0-cp312-cp312-emscripten_3_1_58-wasm32.whl

i.e., reflecting the Emscripten version in the ABI tag. This has now changed to pyodide_20XY_Z to reflect the current year and build number.

Pyodide-powered interactive documentation

Example: NumPy

How? jupyterlite-sphinx

A Sphinx extension providing a set of utilities to convert doctest-based examples to notebooks

provides a Pyodide kernel for those notebooks, to install WASM wheels

+

Simply add the .. try_examples:: 

reST stub to your example's docstring:

>>> x = 2
>>> y = 2
>>> x + y
4

and a button in the HTML source is generated for you!

pip install "jupyterlite-sphinx" "jupyterlite-pyodide-kernel"

Another kernel, jupyterlite-xeus is available as well, which works with emscripten-forge recipes.

Let's try it out!

Future work for Pyodide

  • What we are planning to do
    • Build smaller wheels for bandwidth preservation
    • Packaging improvements
    • WASM-specific optimisations
    • Adapt interactive documentation for more Scientific Python projects, and write a Scientific Python Enhancement Proposal (SPEC) document
    • On Pyodide itself: build farms for Pyodide to improve our CI

Maintaining Pyodide 🧯

  • A small active team of just three people
  • Overwhelming nature of users, requests, packages to support
  • So many areas to contribute:
    • CPython compatibility
    • Better toolchain support
    • Improving adherence to WASM conventions
    • JavaScript codebase
    • Social media
    • Improving community relations
  • So much work!!!
  • So much fun!!!

                                                 is so important

Lesson III

Software sustainability     

Software sustainability

  • Corporate funding for open source helps in most cases
  • The notion of open source being critical infrastructure
  • Organisations also get benefits via corporate social responsibility
  • Becoming a maintainer myself helps me realise the efforts undertaken by other maintainers

The ever so cliché XKCD 2347, yet holds true always!

Working on OSS, full-time @ Quansight?

  • Working with talented folks from all around the world at the forefront of Scientific Python libraries
    • Getting mentorship while on the job
    • Makes you understand that it's okay to be scared when you start
    • —Sounds inconsequential, but step back for a moment and think again: how cool is that!?!
  • Getting to contribute to software alongside people you've idolised
  • Leaders, not bosses
  • On the software front: direct contact between library users

Monte Isola, Italy

Takeaways

  • The Pyodide distribution: https://github.com/pyodide/pyodide (we're open to contributions!)
  • Intersectional lessons from the open source ecosystem
  • Open source software is multi-faceted and complex
  • Software ecosystems need funding and maintenance to be panaceas for an ailing world and thrive
  • Playing your part in the ecosystem, how so ever small it is, helps

Thank you for your time!

Please feel free to say hello!

in/agriyakhetarpal

agriyakhetarpal

agriyakhetarpal

agriyakhetarpal [at] outlook [dot] com

Content licensed under the CC-by-SA Attribution-ShareAlike Version 4.0 International License

Special thanks to

Ralf Gommers

Albert Steppi

Melissa Mendonça

Matthias Bussonnier

Robert Hood Chatham

Gyeongjae Choi

Loïc Estève

...and to many more in my journey

Emoji by openmoji.org

These slides

Pyodide

 @agriyakhetarpal@fosstodon.org 

@agriyakhetarpal.bsky.social