I recently had this exchange with somebody on Twitter.
I've written about Twitter before.. I don't feel any responsibility to follow back when I'm followed. Time is finite resource. Tweets in my feed are essentially calls for attention. It's about 500 people (in my case) knocking on my door asking me to engage somehow, either by reading and caring, or clicking, or whatever. They say fences make good neighbors. In the case of Twitter, your profile page is the fence. I do check out the profile of every person that follows me. If I see something that gives me a reason to believe I'll be educated or even just entertained by following back, I do so. Honestly, it's not a high bar. However, I probably spend about 30 seconds deciding.
I assume anybody that follows me is getting something out of it. If your primary reason for following was to get another follower, you'll probably be disappointed. If you are looking for the behind-the-scenes info on NCIS-LA you will most certainly be disappointed. If you are hoping to be entertained or learn something maybe I'll live up to expectations, or maybe I won't. I don't pay much attention to my Twitter stats so I'll probably not even notice if you unfollow.
Also, with the new group DM feature in Twitter following 50,000 people is just asking to get spammed via Twitter direct messages. Following back everybody that follows you devalues the very purpose of Twitter in my mind. With that much noise in your Twitter feed you are bound to miss the actual good stuff. You may pick up on some good stuff too, but I'm pretty sure the signal-to-noise ratio is going to work against you, not for you.
So the tl;dr version - don't feel obligated to follow back on Twitter.