The key to cleaning comic books is to be gentle. You could also place the comic books in a plastic sleeve to protect them from future dirt and damage. And always store them in a dry, cool place to prevent mold or warping.
For cleaning comic books, first, make sure your hands are clean. Then, use a clean, dry cotton swab to remove dirt from the corners and crevices. If there are stains, you might try a mild cleaning solution specifically designed for delicate surfaces, but test it on a small, inconspicuous area first.