The no 4 or 4x bus from Whitby to Middlesburgh will take you to Staithes, Hinderwell (Port Mullgrave), Sandsend.
The no 93x bus from Whitby to Scarborough will take you to Robin Hoods bay. Just google Arrivabus for timetable.
A short walk to the east side of the harbour will get you on the beach and you can fossil along to Saltwick bay.
Hope that this helps