Once you've built your Custom Audience, you'll have to select it at Ad Set level in the targeting section - it's right above location, demographics and interests. As long as you've selected your specific custom audience it will only target that audience.
If you add other criteria to your targeting, say you add a 20-55 year bracket and men only on top of using your custom audience, then it will target people in your custom audience within that age group and only men. Does that make sense?

Jon Loomer has an excellent library of resource that can help you: https://www.jonloomer.com/tag/facebook-custom-audiences/

If you add your custom audience without any additional restrictions, only people in your custom audience will be exposed to your ads. If you add a demographic, or interests, then FB will require that a user a) is in your custom audience AND b) meets the additional qualifiers you have added.

Custom audiences are groups of users based on lists that were uploaded, events like a webpage visit or conversion and engagement on Facebook. If you upload a list of 10,000 users, you'll be able to show ads to any of those individuals that Facebook can match with their users.

Once you select the custom audience, you can target inside of that group of people by selecting more characteristics. For example, if you targeted people inside of your custom audience that liked "dogs", the list size might go from 100,000 people to 40,000. You would be targeting all the people inside of your custom audience that like dogs.

Hope that helps!