It’s necessary to read your thesis in preparation for the viva. You don’t need to memorise your work, but it helps to have a good recall of the flow of your thesis to make using it in the viva easier.
Annotating your thesis or writing summaries are helpful for viva preparation, and have the added benefit that they have to be done actively. You can’t write a summary without being engaged with your work. You can’t annotate your thesis without being switched on and reviewing what’s there.
You can pass time skimming your thesis, but that won’t help your prep as much as reading actively or investing time in writing about your research. Reading and writing tasks are essential elements of getting ready. Find good ways to do both kinds of work.