You receive notifications when people follow you, but not when users 'unfollow' you. This means in the same time that people follow you (which you're notified for), other followers may have unfollowed you, which would lead your follower count not to increase by the amount that you were notified for. 

People can unfollow for a few reasons:

• People choose to unfollow you.
• People remove their accounts. 
• Batches of spam accounts that are in your followers have been removed from the platform. 

Please note: Due to caching, the actual numbers displayed on follows won't re-adjust until an action is made on your account. An action could be uploading a track, following someone new, or making any edits to your profile.