There is a lovely story from the life of the Prophet Muhammad, remembering that a mysterious visitor came upon him and his companions. The visitor, later revealed to be the archangel Gabriel, proceeded to sit intimately next to Muhammad and quiz the Prophet. He asked Muhammad about three increasingly higher and deeper levels of religiosity, which the Prophet answered sequentially as Islam (wholehearted submission to God), Faith and, lastly, Loveliness (ihsan). This third quality the Prophet identified as worshipping God as if we could see the Divine, and if we cannot, to always remember that God nevertheless sees us.
The sequence is fascinating, as it reveals that what we think of as Islam (the attestation to Divine Unity, the performance of the prayers, the pilgrimage to Mecca, the paying of the alms tax, the fast of Ramadan) mark only the very first layer -- though the foundational layer -- of religiosity. Above that is faith, and above faith is the spiritual and mystical layer of spiritual beauty, for ihsan is literally the realm of actualizing and realizing beauty and loveliness (husn), of bringing beauty into this world and connecting it to God, who is the All-Beautiful.
Throughout Islamic history, this realm of ihsan was most emphatically pursued by the mystics of Islam, the Sufis. Historically, this mystical realm of Islam formed a powerful companion to the legal dimension of Islam (sharia). Indeed, many of the mystics of Islam were also masters of legal and theological realms. The cultivation of inward beauty and outward righteous action were linked in many of important Islamic institutions. In comparing Islam with Judaism, the mystical dimension of Islam was much more prominently widespread than Kabbalah. And unlike the Christian tradition, the mysticism of Islam was not cloistered in monasteries. Sufis were -- and remain -- social and political agents who went about seeking the Divine in the very midst of humanity.
After the Prophet Muhammad, many of the most influential of all Muslims were and remain mystics. Mawlana Jalal al-Din Balkhi, known to Turks as Mevlana and to Americans as Rumi, remains the most beloved of all Sufi poets, whose Masnavi was perhaps the only work ever compared directly with the Quran. Ibn 'Arabi, the Spanish Muslim sage, remains the most widely read metaphysician, and his school of "Unity of Being" (Wahdat al-wujud) has been both influential and controversial from Spain to Indonesia. The most important Muslim theologian, al-Ghazali, identified the realm of Sufism as the highest Islamic quest for knowledge, one that dealt most directly with other-worldly matters.
Nor was the practice of Islamic mysticism limited to intellectuals and poets. At the level of popular practice, some of the Sufi shrines received as many (if not more) annual visitors that the Mecca does for the Hajj pilgrimage. Entire Muslim-majority regions (Iran, Turkey, South Asia, Sub-Saharan Africa, etc.) came to develop understandings of Islam that are and remain inseparable from mystical understandings of Islam. Much of the higher dimensions of Islamic aesthetics (calligraphy and poetry) have been inseparable from Sufism.
And yet, today, the word "Sufi" is a highly suspect one for many modern Muslims, and even thinkers and preachers whose frameworks and anecdotes are permeated with those of the mystical dimension of Islam eschew the mere mention of the word Sufi, either not wanting to alienate their suspicious audience or not wishing to "erode" their authority by connecting their teachings to anything other than the Quran and the example of the Prophet.
So how did such a powerful and beautiful dimension of Islam come to be viewed with such suspicion by so many Muslims?
The marginalization of Sufism came about through an initially unlikely perfect storm, an alliance of European Orientalists and conservative/modernist Muslims, whose agenda in demarcating Islam from Sufism ironically supports that of certain New-Agey Universalists who sought to extract Sufism out of Islam. Let's explore this somewhat odd association a bit more closely.
The Orientalist scholars (whose approach began in Europe and dominated much of the American scholarly engagement with Islam) based their approach on a study of Islam that privileged "classical" legal and theological Arabic texts from 800-1100 C.E. Of all those texts, the most important ones were held to be the ones closest historically to the "foundational" period. The Orientalists became interested in Sufism very early on, almost as early as their translations of the Quran. They found themselves attracted to the deep beauty and wisdom of Sufi poetry, particularly from Persian. Quite inconveniently for them, they were also committed to a bifurcated view that divided the world into Semitic (Arabs and Jews, characterized primarily by law, monotheism, and dry deserts) and Indo-Europeans (Hindus, Europeans and Iranians, who lived through philosophy, art, mysticism and logic). The Orientalists had no problem thinking that entire blocks of humanity share certain "mentalities" and "temperaments" connected to their languages. Even though they admired the poetry of mystics like Sa'di, Hafez and Rumi, they could not admit that Muslims (who were "Semitic" after all) could come up with such beauty, mysticism and poetry. Therefore, the Orientalists decreed that Sufism must be "un-Islamic" and due to Christian, Persian, Hindu or Neoplatonic "influences" -- anything but Islam, anything but the experience of Prophet Muhammad in encountering God, which is what the Sufis have always claimed as the primary source of their inspiration!
The Muslim conservative/modernists (what we broadly refer to as the Salafi tradtion) came to have a profound distrust of what might be termed "the tradition(s) of Islam," believing that the historical tradition of Islamic scholarship -- and the scholars who had been the authoritative interpreters of Islam -- were increasingly irrelevant to the historical trials and tribulations through which 19th and 20th century Muslims were suffering. They wanted to remain pious and observant Muslims, but believed that the way to return to the "glory days" of Islam was to "return" to the original spirit of vitality and authenticity of Islam, before the influence of "foreign ideas" crept into Islam, sapping its authenticity. These foreign ideas they equated both culturally (the contribution of Persians, Indians, Turks, etc.) and intellectually (the traditions of philosophy, mysticism and all non-scriptural sciences).
The idea for the Muslim modernists was that the remedy for Islam consisted of a textual return "away from the blemishes ... of the later phases" back to "yearning for truth" of the founders of Islam. In this, they found themselves oddly in full-agreement with the orientalists. They came to be suspicious of many traditions of Islamic thought and practice that developed through time, including that of Sufism. Perhaps most polemically, they identified Sufism as having contributed to a corrupt and inward-looking mentality that allowed the colonial powers to dominate Muslims. Throughout Islamic history, particular Sufi ideas and practices (such as the "Unity of Being," certain meditation techniques and commemoration of the Prophet's birthday) had always been contested by other Muslims. It was in this modern and modernist context that the whole of Islamic mysticism came to be viewed with great suspicion as being un-Islamic if not outright anti-Islamic.
So where do the New Agers come into play? It was only in the 20th century that human beings became capable of uttering a sentence like "I'm not religious, I'm spiritual." Historically all religious traditions have had mystical dimensions, and their mystical traditions have arisen within the very depth of each tradition, partaking of its key symbols and emulating the spiritual experiences of its main exemplars. It was in this modern context that a deep and new suspicion of the outward forms and institutions of religion was cultivated, with people who believed that they were on the edge (or already inside) a "New Age" of human consciousness. It was these new Agers who, dissatisfied with their own experiences of Judaism and Christianity, turned "East" to the mystical traditions of Buddhism, Hindu traditions and Islam to obtain the mystical truth that they so yearned for -- without necessarily wanting to adopt the legal and institutional aspects of those traditions. In many cases, the engagements were complicated by colonial politics, as the "eastern" traditions of wisdom were connected to colonized countries that many of the same Westerners looked down upon, even as they were fascinated by them.
So what we have had for the last few decades is a situation of Orientalists and Salafi Muslims seeking to construct a "real Islam" that is untainted by Sufi dimensions, and many new agers seek to extract a mysticism that stands above and disconnected from wider, broader and deeper aspects of Islam.
Yes we have learned that the human yearning for the Divine, for beauty, for love and for loveliness is too deeply engrained in the human spirit to be partitioned off or exiled. Today, many Muslims world-wide are increasingly dissatisfied with what they see as dry as stale bread interpretations and practices of Islam, and want -- and demand -- something more spiritual and more beautiful. They know about the deep spiritual experience of the Prophet Muhammad, who came face to face with God, and they too yearn for their own spiritual experiences.
All Muslims seek to emulate the Prophet Muhammad. The Quran reminds them that if you love God, follow Muhammad. The mystically oriented among Muslims take the emulation a bit more literally: If Muhammad arose to have his own face-to-face encounter with the Divine, they too aspire to rise in the footsteps of the Prophet, to have their own meeting with God. As it was said of the great Rumi, they too want to be "off-springs of the soul of Muhammad."
Omid Safi is a Professor of Religious Studies at the University of North Carolina. He is the Co-Chair of the Islamic Mysticism Group at the American Academy of Religion, and the author of 'Memories of Muhammad: Why the Prophet Matters' (HarperOne, 2009).