Related Posts
Well, almost every marketing organization that has a website has experimented with A/B testing at one point or another. Create two pages, test, and choose a winning page. This is an idea that does not need many words to be explained.
That simplicity is exactly the trap. Most teams enter the game full of excitement, run a few experiments, and stop using the program within just a couple of months.
The ability to learn the mechanism is simple enough for anyone, but what keeps an A/B testing program running has little to do with mechanics.
Splitting a population into a control group and a treatment group is a nineteenth-century idea, formalized in agricultural research and later in clinical medicine. Randomization exists to strip out everything you did not think to measure. When a pharmaceutical trial randomizes patients, it does so because the researchers use random assignment due to their inability to control for diet, sleep, genetic factors, and thousands of other elements.
Web experimentation inherited that logic and added enormous sample sizes. A hospital trial might involve four hundred participants recruited over two years. A mid-sized ecommerce site can push forty thousand visitors through an experiment in a week. It is the size of the experiments that makes online testing powerful, and it is also what creates the illusion that results arrive quickly and require no particular expertise.
Failure one is the availability of hypotheses for a testing program to run. A testing program needs a queue of hypotheses, and most teams exhaust their obvious ideas in the first month. Move the button, change the headline, shorten the form. After that, someone has to do the unglamorous work of reading session recordings, digging through funnel drop-off, and interviewing support staff about what customers complain about. That research is where good hypotheses come from, and it is the first thing cut when the quarter gets busy.
The second failure is throughput. Testing a single hypothesis on a page with modest traffic may take three or four weeks to get an adequate sample size. Run those sequentially, and you get roughly a dozen experiments a year, of which only a handful produce a clear winner. That does not justify a tool subscription, let alone a headcount, so the program gets deprioritized before it ever compounds.
This is where the team either develops the internal capability to execute or outsources the process to a third-party A/B testing service that has the analysts, the designers, and the entire statistical process in place. The choice is usually about capacity rather than knowledge. Marketers generally understand what they want to learn. What they lack is a designer who can build six variations this week and someone who will still be monitoring test four when the quarterly planning cycle absorbs everyone’s attention.
The third failure is the expensive one, because it does not look like a failure. It looks like a win.
Consider what happened at Bing in 2012. An employee proposed a change to how ad headlines are displayed. Program managers ranked it a low priority, and it sat untouched for more than six months until an engineer ran the test anyway. The results were so dramatic that the system marked it as a possible bug. The change lifted revenue by 12 percent, worth more than $100 million a year in the United States alone, and it had been sitting in a backlog because nobody senior believed in it.
That story gets quoted constantly for the upside. The more useful lesson sits on the other side. If expert judgment is that unreliable about which ideas will work, then the statistical apparatus around the test has to be trustworthy, and usually it is not. Evan Miller’s analysis of repeated significance testing showed what happens when teams watch a dashboard and call the test the moment it turns green. In one particular case, the actual false positive probability becomes 26.1 percent, more than five times higher than that claimed by the significance level. Peek ten times, and a figure your tool reports as 1 percent significance is really closer to 5. His advice is unfashionable and correct: fix the sample size before the test starts and stop peeking until it ends.
The practical consequence is that a meaningful share of “winning” variations shipped by undisciplined programs is noise. The team celebrates, implements the change, and the conversion rate does not move. After that happens twice, executive confidence in testing evaporates, and it usually evaporates without anyone identifying the statistical cause.
Because most well-designed experiments will not generate a positive result, the number of tests you run matters more than how clever any individual test is. A program shipping fifty experiments a year with weak hypotheses will beat one shipping eight carefully agonized-over ideas. Failed tests are not wasted either; a variation that loses tells you something concrete about what your customers do not respond to, which is information the competitor guessing at it does not have.
None of this takes the place of actually driving people to the first place. Conversion improvements multiply against existing traffic, so a site still needs to be earning the visit through search, referral, or paid channels before optimization has anything to work with. A 20 percent lift on two hundred monthly visitors is forty visitors. The same lift on two hundred thousand is a business case.
But there’s also a sequencing factor. Testing a checkout flow that only 3 percent of visitors reach will take months to resolve. Start where the traffic already exists, which is usually higher in the funnel than the pages teams instinctively want to optimize.
Few A/B tests have failed due to an inappropriate choice of blue color. They fail because the idea pipeline runs dry, the calendar cannot accommodate enough experiments, and the results get read wrong in ways that feel like success. Fix those three, and the compounding starts somewhere around the second or third quarter, when the accumulated small lifts become visible in the revenue line, and the program stops needing to justify itself. Not many teams get to this point, but that has more to do with patience than A/B testing itself.
Ans: Small packs of 50 mods or fewer generally require 3 to 4 GB of memory to support a couple of users. This amount of memory can be sufficient to avoid lag with fewer mods installed.
Ans: Large packs of 150+ mods or more generally need 8 GB or more, and there are no tricks to do it. A pack of kitchen-sink can take even 10 GB or more when a crowd joins the server.
Ans: No, a bit should be left for the OS. Allowing Java to take all the RAM of your machine will make the game unstable and prone to crashes during operation.
Ans: The price of a smaller modded server will be only a couple of dollars per month; a bigger pack of mods for a community will be around ten-twenty dollars.
