I am from the RingCentral Product team.
The admin has the option to define the way the screen pop behaves. They can configure the page to open in a new browser tab. (This is defined in the admin guide available on our
AppExchange listing scroll down to the "Customization Guides" section where the guides are available. Please refer to the section on "Assign the SoftPhone Layout to Users" in the admin guide). However, the behavior of the page opening in a new window or tab does not work in Lightning due to a limitation form SF. It works correctly in Classic. We will continue to reach out to SF and evaluate ways to resolve this issue.