Deuteronomy 13 is part of the code of laws that Moses gave when the Hebrews were at the border of Canaan, about to begin their invasion.
Chapter 13 deals with there being one single God that the Hebrews must worship.
False prophets who tell Israel to worship other gods (likely with the Canaanite gods in mind here) must be put to death.
If one's relative tells them to follow them in worshipping other gods, they must be accused and put to death.
If a city of Israel, led by bandits or corrupt men, falls into idolatry and tries to lead the other cities down the same path, it must be offered as a sacrifice to God.
The point is that even a prophet, even one's own relative, even a whole city must be removed from the land to keep the nation pure and holy, if they try to lead it into idolatry.
It foretells of the cycle of bullshit that Israel would end up going through after settling in Canaan, as they would do exactly that: fall into idolatry and polytheism.
Muhammad was a false prophet, he was just a lunatic. The prophets talked of here are real prophets, whose prophecies come true and who can do miracles, so I don't think it really applies to Muhammad. He was not a glorious prophet sent by God to test the hearts of Israel (or in our case, the Church, since we are the True Israel) but a warmongering madman who did no miracles and killed Christians and Jews.
It should also be noted that a Jew who converts to Islam does not become a pagan or a polytheist, but he does betray the Law that he is supposed to follow, since the precepts of Islam don't line up so well with those of Judaism (even pre-Christianity Judaism).
In that sense it's kind of funny, because Muhammad is actually an exact inversion of the prophets that Deuteronomy 13 has in mind. The false prophets Israel had to be wary of would be men whose prophecies come true and who can do miracles, but who also tell Israel to go and worship other gods. Muhammad was right to call for the worship of the one and only God who ever exists, the God of Abraham, but he was no prophet at all, only a maniac.