When I first started my businesses, I was in for a bit of a rude awakening, in so many ways. I had "shiny object" syndrome; I bought all kinds of things I didn't really need, for stuff that I could have done less expensively. I could have saved myself a lot of time, money and aggravation by being more judicious about what I bought and when I bought it.
So I thought I'd create a small list of things that you DO need in your business, when you are getting started:
1. Legal Paperwork, LLC, Bank Account, Operating Agreements. These things are getting lumped together into one item, but do yourself a massive favor, and get your business set up properly before you do anything else. That means incorporating, getting a tax ID for your business, and making sure you have a separate bank account for your business.
2. Insurance. One of the hardest things to find for a business is insurance. You'd be surprised at just how hard it can be to get proper insurance. You need to make sure that you have Workman's Compensation if you have employees, professional insurance, and possibly an umbrella policy in place. In fact, if you don't have these first two items in place, I would hazard a guess that you don't need to be in business. The state of North Carolina has started cracking down on insurance and Workman's Comp insurance, in particular. You can face tremendous fines if you don't have proper Workman's Comp in place for your business.
4. Some sort of internet presence. You can decide what sort of internet presence you'd like to have. You may decide to go ahead and register a domain for a website, but this isn't strictly required. Set yourself up a Google Plus page or a Facebook page so that you are searchable on the internet. A website is obviously a nice thing to have, but when you first start, a website isn't a requirement.
5. Business cards. Whether or not you absolutely need business cards will depend on the exact nature of your business. Most people have some sort of brick and mortar component to their business, in which case a business card is a requirement. You can get business cards printed very inexpensively through online print companies like Vistaprint. You can hand those out until you get something else prettier made.
6. Phone and phone number. This should go without saying, but I'll put it out there anyway. You don't need anything fancy, but you should have a phone with good reception and a nice voice mail message. You will need to answer your phone whenever it rings, and return calls promptly to get your business up and running.
7. A good accountant, bookkeeper or accounting software. Keeping accurate records in your business is critical. If you don't, you will spend a lot more time trying to recreate things and pulling the expenses out of your personal finances than you will believe possible. Make sure for your own sake that you don't mix finances if you can help it at all.
Not part of the business, but equally important, is having a good support structure in place. Make sure that you have a great support structure of family and friends in place for those times when nothing seems to work the way that it should.
There are many things you don't need, and I'll start a quick list of those here. You don't need an expensive website, office, logo, branding, letterhead, or other items. You don't need expensive CRMs, computers or phones. You just need something that's good enough to get you up and running. There will be lots of people who will find you and offer you all sorts of things for your business that you frankly don't need. You can spend many thousands of dollars on things that your business doesn't need, and that don't work particularly well by falling prey to this shiny object syndrome. Resist the urge, and save your pocketbook.