A ‘User Group’ (found in the Admin Setup "Manage Groups" menu) is primarily used for sharing Documents, Articles and Courses. You are able to create groups that will have access to specific folders and documents based on what you configure, then add the desired users to it.
A ‘Security Role’ meanwhile is used to control the permissions for which areas in the system and what configurations users will have access to see and/or edit when logged into your portal. When you assign a Security Role to a user, that user will only have access to the entities and fields that have been configured with the appropriate Read, Create, Edit, and Delete permissions.