It's not really harmful as it's decomposed material and bio-film. However, it will eventually reduce the flow through the filter so periodically, hoses need to be cleaned out.
Its definitely worth doing - the gunk can be a big source of nitrates over time. I've been really bad at keeping up with it in the past but now I've bought a pipe cleaner I'm going to do it every couple of months. In the past I've had opaque pipes but now they are a bit more translucent I can see how much gunk builds up in them.