
The problem
Machine-learning pipelines spend much of their time in preprocessing: encoding categories, imputing missing values, building features. Doing it in pandas and scikit-learn is familiar but slow on large tables, and the fitted pipeline then has to be rebuilt for production.
What Gators does
Gators gives preprocessing a scikit-learn-style fit / transform API on top of Polars:
- 131 transformers in 12 categories: encoders, imputers, scalers, binning, feature generation, feature selection and more.
- Lazy execution. Transformers accept Polars
LazyFrames, so a whole pipeline can be optimised before it runs. - ONNX export. A fitted pipeline exports to ONNX for low-latency serving.
Results
On 500,000 rows (Apple M3 Max), compared with scikit-learn:
| Transformer | Speed-up |
|---|---|
TargetEncoder |
15.0× |
OrdinalEncoder |
9.1× |
OneHotEncoder |
6.2× |
My role
I created Gators at PayPal, where it was open-sourced, and I am its main contributor and maintainer.
Get started
pip install gators # or gators[onnx] for ONNX export
The documentation covers every transformer with examples.