P—07 Case study · Research
MedTech — Clinical Feature Research
Role
ML Engineer
Status
completed
Started
Mar 2026
Last activity
Oct 2026
Stack
7 technologies
MedTech — Clinical Feature Research
Python · PyTorch · scikit-learn
Overview
Can patient questionnaires predict pain and tumour stage? The data is 226 patient records, studied across five notebooks. The first explores the SF-36 quality-of-life survey with PCA and clustering. The other four train neural networks that predict TNM stage and pain together, then separate models for T and N stage, then classical and ordinal models, and last repeated cross-validation and PCA comparisons.
Three feature sets are compared: 54 scored answers, 220 answers coded as Vata, Pitta or Kapha, and all 256 together. Every set also includes the tumour site. Pain predictions reached R² 0.670 with all features and Ridge regression, and 0.619 with the scored answers and ElasticNet. The Vata/Pitta/Kapha set alone did worse. N stage predictions beat simple baselines by a small margin. T stage predictions did not beat them consistently.
These results are early. Some runs duplicated records before splitting train and test data, others scaled the whole dataset before splitting, and some questionnaire items may overlap with the pain score. The private data was not rerun during cleanup. Phase 2 will recover the missing runs, the questionnaire scoring rules and the outlier predictions.
Outcomes
Patient records
Research notebooks
Best pain R²
Saved confusion matrices
Impact
- 01Built five research notebooks, from an SF-36 survey study to neural, classical, ordinal and PCA models that predict pain and tumour stage.
- 02Compared three feature sets on 226 patient records: 54 scored answers, 220 Vata/Pitta/Kapha-coded answers and all 256 together.
- 03Reached R² 0.670 for pain with all features and Ridge regression, and 0.619 with the scored answers and ElasticNet, on one 80/20 split.
- 04Kept all 112 confusion matrices and all 1,323 original numbers from the results workbook, each linked to the run it came from.
- 05Added baseline comparisons and repeated cross-validation, and listed what is still missing: some source runs, the scoring rules and the outlier predictions.
Features
01
Three feature sets
54 scored answers, 220 answers coded as Vata, Pitta or Kapha, and all 256 together. Each set includes the tumour site. Testing tumour site on its own and checking the answer codes come next.
02
Neural networks
The first network predicts TNM stage and pain together. Later versions split it into separate T, N and pain outputs. Small, big and ultra network sizes are compared.
03
Classical and ordinal models
Standard classifiers, ordinal classifiers that respect the order of stages, and regression on stage numbers that rounds or thresholds the result back to a stage. Pain prediction tries five model families.
04
Repeated cross-validation and PCA
T stage is predicted from the Vata/Pitta/Kapha set and N stage from all features, each with 5-fold cross-validation repeated 3 times. PCA, fitted on training folds only, compares 10, 15, 20 and 30 components with a 95% variance cut-off.
05
Research archive
Five notebooks, a results workbook, summary data files and 112 confusion matrices hold every recorded result. Cleanup did not recompute any of them.
06
Known limits and Phase 2
Early runs duplicated records before the train/test split, and later runs scaled the full dataset first. Older spread figures were pooled, so they do not show variation between folds. Phase 2 covers these gaps.
Architecture
The data comes from two private CSV files, one for each round of the project. The SF-36 notebook separates the survey outcome from the predictors and drops identifying columns. Feature building, missing-value filtering, filling in missing values and scaling are all fitted on the training data only. Rebuilding the score from the questionnaire is a separate comparison. The four later notebooks use the final question columns, clean up the TNM stage labels and build the three feature sets: 54 scored answers, 220 Vata/Pitta/Kapha-coded answers and 256 combined, each with the tumour site. The first models predict one of 21 TNM combinations plus a pain score. Later models predict T1 to T4 and N0 to N3 separately. Their neural networks duplicate records only inside the training data, unlike the early joint runs. Classical and ordinal models are tested on an 80/20 train/test split. The later T and N models use 5-fold cross-validation repeated 3 times, are compared with simple baselines, and turn a predicted number back into a stage by rounding or by thresholds. PCA is fitted inside each training fold. Earlier runs scaled the data and encoded tumour site on the whole dataset first, which limits those results. Pain errors are on a normalised scale. The checked results go into a workbook, a JSON archive and the Phase 2 list of missing runs.
Stack
Development log
· Milestone
Added four notebooks and a results archive
Added four notebooks: TNM stage and pain predicted together, T and N stage predicted separately, classical and ordinal models, and repeated cross-validation with PCA. Kept 112 confusion matrices and a workbook linking each result to its run. Pain predictions are better with the scored answers or all features. N stage edges past simple baselines and T stage does not. Phase 2 lists the missing runs.
· Started
Early-stage implementation of multiple ML models done.
Early-stage implementation of multiple ML models done. It is observed that signal is inherently leading to poor correlation between prakriti and patient's cancer stage. Further study with temporal data could prove to have an effect.