First: build a server that is capable of virtualization - XenServer works well.
I use OpenIndiana with Zones, but that's less recommended unless you're a real Solaris junkie. Use something to make sure that your VM storage is redundant - it doesn't necessarily need to be fast. Raid 1 of two drives or Raid 5 works. Use a small SSD for your OS/hypervisor dom0 (32GB+).
Second, handle your network. Choose whatever router works for you. Get a multi-wan router if you feel like snapping more network connections on. Ubiquiti's EdgeRouter works wonders if you're comfortable with vyatta/linux/iptables and want to QoS your family/friends/roommates torrenting down to a quiet level. Cisco's RV042G will 0wn any VPN and VLAN setup you decide to build.
For the most network buttons and switches (and to support cool stuff like VLANs for QoS and guest segmentation), grab one of Cisco's SOHO switches - SG200-08 or SG200-16. They are managed switches that are not ridiculously expensive. TBH, the Cisco SOHO series of switches is really all I'd recommend if you decide that you want managed switches. They also handle really awesome network stuff like LACP/link aggregation if you want to go completely nuts.
Next, handle your wifi. You're going to want this to be rock-solid or you're going to be sadpanda. Ubiquiti UniFi owns. Or open-mesh (but that has an online dashboard :( ). Or the $1000-per Cisco AP solution, but that will waste your life savings. If Ubiquiti UniFi, build a base VM (or run Waratek's Java PaaS) for the UniFi controller. The base controller comes with an installation of MongoDB, but you can build your own somewhere else if you want. You can buy these in three packs at Streakwave.
Next, Ubiquiti mFi for at-the-outlet power monitoring and automation, and environment sensing. You can get data from this and turn stuff on and off at the outlet level. You can also build schedules, device relationships, and scenes. You're going to need to install the mFi controller on it's own VM on the server (or in the Waratek PaaS if you got one - it's Java+MongoDB, also).
Additionally, if you're really bored, you can build your own mFi firmware that includes things like SNMP for monitoring power with something else (Think StatsD for your house...heck, you could even use StatsD if you really wanted to.).
Lastly, automation at the light level - "Connected by TCP" is one brand for basically-just-LED-lights that have an API and monitoring and it's available at Home Depot most of the time. You can turn them on and off and they're white light for not ridiculously expensive. The next option up is Philips Hue. Those let you turn lights on, off, change colors, blink to the music.... The response time of all of these bulbs is a little weird, but you can still do some awesome stuff with them.