Now's the time to use your little brush and get into cleaning. Number one rule is being gentle and not using too much pressure. If you have a microfiber cloth that might come in handy as well.

You can easily take that black rubber thingy off your fan to get into all the corners with your brush (I hear they call that thing 'rubber fan bumper' - what a word).

When you've loosened the dust, just take the hair dryer again and blow it away.