A protocol is a method of communication between two parties. In the case of RuneScape, the protocol is the method of how the game client and server communicate. There are a number different protocols in use in the RuneScape game, each with a different purpose.
In addition to these, Jagex will also have protocols for communication between login and world servers and such. However, since these are internal, we cannot document them here.