You are part of the way there...I will try to give you an example of how it can be done...
Following your "custom layout" creation for a unique colored category-
you will see that each frame of your layout has a small pencil/wrench looking icon in the corner.
This gives you further access to change the style of the frame.
If as example we wanted to change the background the frames to blue, we can do so by entering "background: blue" into the CSS in the "Frame Style" field.
Of course you may want to take this a bit further for each block of that layout (ie blue banners, red text, green links, etc...
This can also be done by entering the CSS needed within the "Block Style" of the blocks Appearance Tab.