chore(deps): update dependency colbert-ai to v0.2.22 #14
Loading…
Reference in a new issue
No description provided.
Delete branch "renovate/colbert-ai-0.x"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This PR contains the following updates:
==0.2.21->==0.2.22==0.2.21->==0.2.22Release Notes
stanford-futuredata/ColBERT (colbert-ai)
v0.2.22: ColBERT 0.2.22Overview
Bug Fixes
rflag for regex #368.Improvements
The main fix is #390 which fixes the AdamW import error (
transformers.AdamWis deprecated). This allows you to use the latest version of transformers. However the latest transformers depends ontorch>=2.1but ColBERT currently depends ontorch==1.13.1. As shown in the sections below, using the latest version of torch (2.8.0) and transformers (4.55.0) does not break core functionality but does result in different indexing, retrieval and training results than usingtorch==1.13.1andtransformers==4.38.2(the last version that hadAdamW).ColBERT 0.2.21 and 0.2.22 Comparison
In this section, I'm going to compare the index artifacts, retrieval artifacts, and training logs between two different
colbert-aiinstallations:torch==1.13.1andtransformers==4.38.2(the last version withAdamW) referred to as "0.2.21" below.torch==2.8.0andtransformers==4.55.0(the latest versions of both), referred to as "0.2.22" below.My indexing, retrieval, training and comparison scripts can be found in my colbert-maintenance repo
ConditionalQA Index Artifacts
I compared the shapes and values (
torch.allclose) of the.ptfiles in both indexes (of the UKPLab/DAPR/ConditionalQA document collection). All but one pair of tensors have the same shapes (it's an important one:ivf.pid.ptwhich maps document token embedding IDs to centroid IDs) and none of the tensors' values passtorch.allclose:The changes in from 0.2.21 to 0.2.22 result in nearly wholesale index artifact changes.
ConditionalQA Retrieval Metrics
I compared aggregate and query-level metrics as well as raw retrieved passage IDs between the two
colbert-aiinstallations.The changes in PyTorch (
1.13.1-->2.8.0) and Transformers (4.38.2-->4.55.0) result in a decrease in Mean Recall@10 and increase in Mean MRR@10. On average there is 1 different passage retrieved per query between the two versions.Here are the query-level differences in metrics, where "Increase" means the metric increased in the 0.2.22 install (relative to 0.2.21).
In most cases the query-level metrics are equal for 0.2.21 and 0.2.22. There are 4 queries for which Recall@10 decreases and 8 queries for which MRR@10 decreases for 0.2.22.
Training Dynamics (MS MARCO)
I trained the default
bert-base-uncasedmodel for 1000 batches withcollection.tsv,queries.train.tsvandtriples.train.small.json(JSON file created from the available tsv).0.2.22 results in a higher training loss, higher mean positive document score and higher mean negative document score.
Multi-GPU Setting
I also ran indexing, search and training for the two
colbert-aiinstalls on 4 x L4 GPUs. For indexing, the results are similar---all but one index tensor's shapes match, none of the index tensor values match.In the case of multi-GPU search, the Mean Recall@10 and Mean MRR@10 both decrease for 0.2.22:
At the query-level, multi-GPU search results in a net of 2 lower MRR@10 values for 0.2.22.
For multi-GPU training, 0.2.22 results in a lower training loss, and similar mean positive and negative scores.
Backwards Compatibility
Installing
colbert-ai[torch]==0.2.22withtransformers==4.38.2yields identical index, search and training artifacts ascolbert-ai[torch]==0.2.21withtransformers==4.38.2.In conclusion, while using the latest PyTorch and Transformers versions yields different index artifacts and thus different retrieval results as well as different training logs, the core functionality is not broken in 0.2.22. With this deprecated AdamW fix, users can now install
pip install colbert-aiwithout error, install the latest transformers and torch version, and use the core functionality. The differences between index, search and training across different PyTorch versions will be investigated and documented beforecolbert-ai's torch dependency is changed to 2.x.Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about these updates again.
This PR has been generated by Renovate Bot.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.