The EPA recommends that vermiculite insulation remain in place and not be disturbed for the potential of being contaminated with asbestos.
Disturbing the insulation could cause the asbestos fibers to become airborne which increases the chance of exposure to asbestos in turn increasing your risk of developing lung diseases including asbestosis, lung cancer, or mesothelioma.
There are inexpensive DIY test kits that can be purchased to test for asbestos, which if it comes back negative there would be no issue with removing it. However if it is found to contain asbestos, it is not recommended to remove it yourself and should hire a professional asbestos contractor.
Another issue you may run into with removing the vermiculite on your own is the city dump may not accept it, or limit the quantity as it is considered to be a hazardous material.
More information about vermiculite insulation can be found on the EPA's website here.