You aren't alone. 

There is an ongoing serious problem with subscription numbers which appears to be affection many, many people.

Here's an active link talking about how subscriber losses are offsetting subscriber gains…

Sharing and Connecting on YouTube

"We aren't just losing invalid or idle account SUBS - There's a glitch; I've proved it. 

https://groups.google.com/a/googleproductforums.com/forum/#!category-topic/youtube/sharing-and-connecting-on-youtube/V4DPLGCzp84