The Tanist (or Tannist) is an interesting concept, and not much has been written about it. In its simplest terms, a Tanist was a royal successor. Tanistry seems to be Celtic in origin, and appears to have been imported into Scotland from Ireland in the fifth century. In the earliest days, the Tanist was not necessarily directly related to the king, or even the same branch of the royal family; however they would share a common ancestor. In fact, during the early middle ages, the King was elected by the noble princely families, and the Tanist was elected as well. It was a lifetime post.
In theory, the Tanist would have been an ambitious and capable successor, “without blemish”, able to take on the rulership in a time when a chieftain’s life expectancy often did not allow for his sons to achieve manhood. The Tanist Succession would encourage rotation between branches of a family, and was considered a fair way to keep balance. However, more often than not, it led to dynastic infighting.
Malcolm II, in 1005, was the first Scottish monarch to introduce hereditary monarchy and female-line succession at the same time, since his heir, Duncan II, was descended from his eldest daughter. This innovation caused great conflict and he had to spend many years clearing the way to the throne for his grandson.
In fact, Grouch (known as Lady Macbeth) was descended from the rightful King Kenenth III, who was killed by Malcolm II. Then Grouch’s father, Boede, recognized as the logical Tanist of his branch, was also killed by Malcolm II. In 1032 Grouch’s first husband Gillecomgain was killed by Malcolm II in an attempt to get rid of her, but she was elsewhere when her husband and 50 men were burnt to death in his fortress. No wonder Lady Macbeth urged her second husband to kill Duncan!
So the concepts of Tanist Succession and Patrilinear Succession bumped into each other and wreaked havoc for centuries until Tanistry was abolished by James I (James VI of Scotland). The system lingered in a diminished form in Ireland until the mid-19th century.