Kirsten, yes, it was the dry Nov/Dec/Jan/Feb that shook up a lot of the fruit. How old was the tree when you planted it? Quite small? Five years in the ground for a 2 year old tree isn't very long for a fig, so it can still be vulnerable to swings in the weather.
Where are you as far as heat; inland or by the coast? Nights are very chilly by the coast still. My figs are always delayed without heat.
Did you give the fig lots of compost and rotted manure last fall? Mulching it deeply out to the dripline is important all year long. That helps save water and improves the soil, helps suppress weeds.
If there aren't very many figs, and they aren't even worth cooking with, you might try taking them off, water the tree out to the dripline, feed it organic fertilizer high in the middle number, phosphorus, and it might go again. Mulch it with leaves and mowed grass to at least 4 inches deep, maintain it at that depth.
If it doesn't try again, then giving it the fall treatment ought to improve next year's crop.
Love your soil.....feed your worms...
(Used to be Sweetpea, contributing here since 2002)