I don't want images to be pixalated when someone clicks on them, so I want to decrease the size that the lightbox modal for images is right now (1080px) to perhaps 600-800px. How can I do that?

You can set XLarge image width in K2 Parameters -> Images setting or do it in the Category->Item image options->Xlarge image width

I don't see any change. The modal window has the same size..
It only gets smaller if I decrease the size of the browser window (which is reasonable).

@YannisDr

The problem is not with the modal. The problem is that you have set high dimensions for resizing in K2 image settings and you are uploading small images. The modal opens the image at it's native size. Only when the image does not fit in the browser resizes it.

Probably that's the case. Thanks for noting this down.