We are finally here. Over the last few weeks I’ve spent some time setting this post up a bit. In week 1 of this series I talked about getting your soundgrid network setup and addressed a few things you should do to increase network stability that waves doesn’t talk about. Then, last week I talked about monitoring your network and the tools that you have available to do that. This week I’m going to talk about properly identifying your issue and then write a bit about troubleshooting in each arena (networking issues, clocking issues, server issues). Let’s get started!
I bet the first question you’re asking is how you know what exactly the problem is…well, after last week you’re actually monitoring your network a bit. The best indicator I can give you is to watch processing load. This gets a bit harder on the LV1 but at least within multirack there are a few places you can watch this. I love having the system monitor window open so I can keep an eye on the meters. What you’re looking for is consistency. If you’re seeing your load bar jumping erratically, perhaps with large percentage hops (both up and down), you might have a server issues. From that same system monitor you can monitor network usage and again you’re looking for consistency. If you have a network issues, you’ll like have huge hops up and random dips followed by an overload. Running in LV1 this is harder to monitor and look for but you can watch the windows “task manager” and keep an eye on the network tab. Lastly, if both of the networking and cpu usage seems stable, it’s probably time to check on your soundgrid clocking setup. It is important to note that networking and clocking issues look pretty similar so that’s why I’m addressing both this first week.
In fact, that’s why I’m starting with networking/clocking issues because the vast majority of soundgrid networking issues are an issue with the physical network (either the setup or gear related) or with clocking issues. So let’s dig into the network side of this problem. Before you dig too far into testing, be sure that you are using certified gear for your soundgrid network. Networked audio, especially at this scale, is extremely fussy. Waves has taken the time to test and publish a list of gear that will work without issue (assuming it’s properly functional). After you’ve done that than it’s time to start messing about. The first step is to systematically check the hardware/cabling involved. I usually with start with a simple cable tester and about twenty minutes of cable testing. Since we are supposed to be using STP (shielded twisted pair) I’ll also go grab a multimeter and check the shield for conductivity to make sure that is still effective as well. If that all checks out, I’ll see if this perhaps is a switch issue. Now unmanaged switches (the majority of soundgrid approved switches are unmanaged) are extremely stable and usually work like a charm but sometimes they just need to be power cycled and the cables moved to another port (this exercising the jacks and connectors). I know, it seems stupid to just switch ports or unplug and plug in but I can’t tell you how many times I’ve solved an issue just by unplugging and plugging a cable back into the same jack. If you’ve done all that you may need to find an IT buddy and do some packet sniffing to see what’s happening. In about 5 years I’ve actually had a switch and an intel PCIe network card fail so don’t be afraid to get aggressive with your testing. Typically issues will show up in windows as well. When my network card was dying I noticed by soundgrid connection was no longer a gigabit connection, it had been slowed to just 10/100. You can find this by going to your adapter settings in windows, than click on your soundgrid network adapter and choose, “status.” Make sure it still reads 1 Gbps.
If you’ve stepped through the basic network checks and you’re still having issues, or worse, audio dropouts (the most visible indicator of issues) be sure to start with checking out the verbose log we saw talked about last week. See if there is anything of note, but at this point you may be looking at a soundgrid clock issue. I don’t have the time to explain what audio clocking is but I found this great and brief article to summarize what clocking is and why it’s important. If you’re running and LV1 system you need to check and make sure one of your I/O devices has been set to the master clock and the others are slaving from that. In a multirack setup, you need to clock to the console your inserting your plugins into. With waves you have three options to clock your system digital (clocking through the madi cable), external clock (i.e. wordclock), and what waves calls SoE (Soundgrid over Ethernet, allows waves devices to clock from other devices on the soundgrid network, you’d use this with waves cards inserted in your console). In my case I’m using and MGO (different version of the MGB) and when I set it up I was clocking digitally. But over time issues developed here (either on how SSL handles madi or within the firmware of the MGO) that made me switch to externally clock the MGO with a wordclock signal from my console. I was fortunate enough to have a wordclock output on my console that allowed this easy fix. I say all that so that if you jump down the networking hole and don’t find anything, verify your clocking setup. I chased what I eventually learned was a clocking issue for over 4 months while I was experimenting with latencies, cabling, switches, etc when all it was was a clocking issue with soundgrid. After discussing this issue with waves tech support, the tech I spoke with mentioned that externally clocking is the preferred clocking setup. I’m not sure if that’s in the manual but if you can externally clock your I/O with your console via a vehicle like wordclock, you should. If you’ve inserted a waves card into your console, your network will run on SoE so you just need to make sure any devices are correctly slaving to that card. You’ll see an SoE settings blank to tell you that status of your setup.
The last area you can have issues is within your server. The waves servers are incredibly sound and bulletproofed before they are shipped. But if you are seeing random overloads in your server you’re likely just hitting it too hard. Some types of plugins (i.e. reverbs, delays, etc) draw on the cpu a lot less consistently so it’s important to watch your usage over time not just after the plugin loads to make sure your server can handle the load. If your usage is in the yellow, consider your server maxed out. If it hits red, you’ll likely start experiencing dropouts simply because the server can’t handle the work. But, the are a few ways around this. If you’re in a live setting you’ve likely lowered the latency settings down to the lowest possible to reduce the travel costs of the external processing. If you just need that extra bit of processing, consider dropping the latency down a step or two. If you’re using multirack live with an MGO or something similar, you just need to the adjust the network latency settings. The driver latency only applies if your setup requires the use of the soundgrid network driver (which usually only applies to soundgrid studio usage). One or two ticks up in this menu can get you another 15-20% processing capability and still be unnoticeable in a live environment. Some experimentation required as to what feels good in your room but there is flexibility here. The goal is to find a balance that works in your room for your setup.
If you’re still having issues, don’t be afraid to call waves tech support. The wait on the phone can be a bit long at times so be ready to be on the phone for a bit. Sometimes you’ll hang up the call and be a bit frustrated with the answers you’re getting but I’ve never ended a call without something I can try if not a solution. They can also remote into your soundgrid controller and take a peak at your setup. I’ve found this to be particularly helpful. Note that they have a different number to call for live sound issues. You can always call the main line but they try to prioritize live sound issues a bit higher than studio issues because of the time sensitivities involved.
Well that’s it for this week. I hope you’ve learned something. If not, or if you have some questions, please leave me a comment below or on facebook. I’d also love to follow up with you if you email me at daniel@studiostagelive.com. Next week we’ll wrap up the series with a short post of some of the tips and tricks I’ve picked up over the few short years I’ve used waves in a live setting. Be sure to sign up at this link if you want to be notified when that post goes live. See you next week!