You receive notifications when people follow you, but not when users 'unfollow' you. This means in the same time that people follow you (which you're notified for), other followers may have un-followed you, which would lead your followers count not increasing by the amount that you were notified for. 

People can un-follow for a few reasons:

• People choose to unfollow you.
• People remove their accounts. 
• Batches of spam accounts which are in your followers have been removed from the platform. 

Please note: Due to caching the actual numbers displayed on follows, won't re-adjust until an action is made on your account. An action could be uploading a track, following someone new, or making any edits to your profile.