This is usually caused by water constantly being left behind in the folds.

One way to deal with it is to dry the seal every time after use (a bit of a faff) or wipe regularly with clear vinegar.