Contrary to many duct cleaning services, mold is pretty rare in the actual duct work.
What it sounds like is that you do not have an adequate vapor barrier installed properly on the ground and that your ducts are pretty leaky. (I take it they are in the crawl space).
- What happens is when your furnace or AC kicks on, it is set to deliver X amount of conditioned air through out the house.
- In order for this to happen it needs an air source which typically would be from your return grill.
- If the return is too small or there is a hole in the line it will pull the air in from elsewhere which includes the clay smelling air in your basement.
Another possibility is while trying to deliver this air it could also be blowing out in your crawlspace and since warm air likes to rise it is simply coming in seemingly from everywhere into your living space.
The final possible issue that is probably exacerbating this is air leakage from your crawl space into your home and from your home into your attic.
I would suggest you get an energy auditor in there to perform a blower door and duct leakage test to help find what the real issues are and where they are at.