The r group of an amino acid would need to have what characteristic to be located in a cell membrane?

The R group should have the ability to form hydrogen bonds.

The R-group take a lot of different chemical forms and can be acidic, basic, polar or non polar. Amino acids that are either acidic or basic consequently have either positive or negative charge. These types of R-groups are stabilized when surrounded by water.

The plasma membrane is the membrane that delimits a cell. It separates the inside of the cell (the cytoplasm) from the outside environment. It is mainly composed of lipids, particularly phospholipids, between which proteins can be inserted.
An amino acid is a carboxylic acid which also has an amine functional group. so it has both a carboxyl group -COOH and an amine group (for example a primary amine -NH 2 or a secondary amine -NH-).
It is customary to classify amino acids into four groups according to the properties of their side chain: acid, basic, hydrophilic, hydrophobic.
apolar amino acids (hydrophobic) tend to occupy the heart of proteins or offer points of adhesion on their surface. In a protein membrane, it tends to occupy the inner part of the membrane, due to lipophilic (hydrophobic) property of their R group attracted by the membrane's lipid. 
polar amino acids (hydrophilic) tend to occupy the surface of proteins. They can be acidic, basic, or neutral. They have a strong chemical reactivity and are therefore found in the catalytic site of enzymes.