Substance on the fretboard rubbing off, should I be worried?

I got this guitar used several years ago, and from some googling I think it’s about 20-30 years old.

I’m restringing it so I thought I’d take this opportunity to post some pics of the neck, cause there’s this weird pattern that looks like a layer on the surface of the fretboard is slowly being stripped. When it’s insanely humid I’ll actually even get residue on my fingers from playing.

I can’t tell if this is some sort of polish that’s being stripped off, or if it’s just dirt that built up before I got the guitar.

Do I need to worry about this?