It's possible it's your mods, try giving it a whirl on a clean game if you haven't already.
It's also likely that it's your graphics card, seeing as how a lot of people complained about similar problems with that graphics card. (GeForce or something, right?) It's also possible you have a virus, run a scan if you haven't. However depending on what kind of anti-virus you may have, it may not list or get rid of most viruses.
Also run a defrag with Disk Defragmenter (now called Disk Optimizer on Windows 10, and I think 8 as well. What a stupid name change, really.), after that, restart the system for the defrag to fully take effect. If you don''t know what it does, it basically clears up useless files (files that do nothing, have nothing, and have no purpose other than to slow your PC down. Basically it slows down your computer from aging. It helps especially when you're picky on what files you want to clear from your computer. It probably seems stupid to bring up, but it helps. Your graphics card could be having a hard time working with a lot of crap stored on your PC, so a defrag might make a difference.
But, it's Nvidia, so it's more than likely it's your graphics card. (If GTA 5 isn't the only game with problems it's either your graphics card or a virus.) I'd recommend using a graphics card from AMD. Unlike Nvidia, they get complaints, yeah, but not as many.
Hopefully I helped you out a little.