What we'll be creating:
A rectangle to oval hover effect. Great for portraits!

1) Select the rectangle and style its hover state

2) Give it a border radius of 50%
When using a percentage border radius, the radius is calculated based on the size of the element. If the width and height of the element are equal, then using a percentage border radius will result in a perfect circle. However, if the width and height are different, the percentage border radius will create an oval shape because the radius is proportional to the size of the element.

Note: a fixed pixel value won't work
A fixed pixel value for the border radius will yield a pill shape no matter the pixel value. See the explanation above.

3) Add a transition effect to smooth out the animation
Select "All Properties" or target "Border-Radius" specifically. Both will work.

Note, this isn't possible in Figma (without getting super hacky?) as Figma doesn't allow for percentage-based corner radii.