Sure, age is going to play a difference. You realize you're not invincible anymore, and you realize things hurt long afterwards. You don't bounce like you once did. Traffic around the bay area here is a nightmare, and drivers are idiots. Road conditions can suck.
Find some good people to ride with who will ride at your pace (BARF is a place to start if you don't know anyone - or PM me, I'd be happy to meet someone for a ride after work every now n then).
Come on out to a track day - I work with PTT (Pacific Track Time) - there are others out there as well, but obviously I'm biased lol. We have a C group classroom all morning and dedicated coaches all as part of your day and can work with you on what you need. Help you overcome some of those fears.
I'm a firm believer, even more so now than I used to be, that technique is key, and the best thing for gaining confidence. After a bad crash a few years ago I was terrified of my bike, braking, and that particular corner (this was on track). Learning technique
and I feel like I've overcome that - there's still a ping in the back of my mind at that particular corner, but it's no longer affecting my speed or ability through that corner (in fact, just recently had a big 'i'm over it!' moment the last time I was out there - felt great!).
So go learn the skills. Learn what your bike can do, train yourself to respond properly. The confidence will come with that.
And, a bit of reservation in this area isn't a bad thing