In love no rival canst thou bear,The first line reflects the theme that is prominent in the Old Testament that God is a "jealous God". Probably the primary sin that Israel struggles with in the Old Testament is idolatry; they were often straying into worship of the gods of the nations that surrounded them. God considered idolatry to be spiritual adultery and would use a combination of enticements and punishments to lure Israel back to pure worship of Him.
But thou art full of tenderest care;
This idea of the jealousy of God seems strange to us because we think of jealousy as a human weakness that sometimes leads to terrible acts of evil. Indeed, there are many people today who think the God of the Old Testament is an evil character because of this jealousy and the punishments that sometimes result. Our mistake, however, is that we are thinking about God in strictly human terms. The idea of God's jealousy is only an analogy to human jealousy. It gives us an idea of the intensity of God's love for us, and that he wants us to exclusively love him, but we must also realize that human loves and emotions are merely a pale reflection of the divine reality.
God's love for us is a desire for what is best for us, and because God made us, He knows exactly what we need and what is harmful for us. The fact is that God is the source of life and being in the universe. Our existence, our lives, and every other good thing we experience ultimately comes from God. Also, God didn't just give us these things in the beginning, but he constantly provides our existence and life. If God ever stopped loving us, we would cease to exist, and this is true for the whole universe in which we live.
If all of this is true, then the best way to live is to seek God with our whole hearts so that we live in harmony with our creator and the king of the universe. Setting our love and life direction on any other thing is foolish because it can never live up to God. A new job, a new house, or a new wife might appear to be what we need for happiness, but they can never live up to what we really need, which can only come from God. Therefore, God's jealousy is for our own good. He knows nothing else besides Himself will give us what we need, so God does everything possible to lure us to love Him, while still allowing us the freedom to reject Him because love without freedom is not really love.