Values obtained in A/B tests are not stable.
complete
Nicky Toma
complete
A
Auke Buma
We have the same issue with the GA AB-tool. We can see the same device, the same unique user, being assigned to Variant 1 and when restarting the app at a later time being assigned to a different Variant.
In other words, during an AB-test, once a player is assigned to a Variant this player should stay in this variant until the AB-test is stopped. What we see happening is that players can get assigned into a different Variant during the test, meaning the data is not to be trusted and basically makes the whole AB-tool useless.
Nicky Toma
Auke Buma: Hey Auke, thanks for reporting! could you please tell us the name of the game this is happening in, or the game ID. Our backend team will investigate.
A
Auke Buma
Nicky Toma: Hey Nicky, thanks for reaching out. I am in the process of getting in touch through other means with your colleagues. So I will await that to go through first.
Nicky Toma
Hello! Thanks for the feedback!
Could you please elaborate a bit? You are selecting a variable to be investigated in the A/B test, and when starting the A/B test another variable than the one you selected is being investigated?
L
LPG
Nicky Toma: Hello! Thanks for the reply!
Yes, it is.
For example.
Key : Key_A Value :Test_A
Key : Key_B Value :Test_B
Suppose you have a variable like this
On the first invocation, Test_A is retrieved, but
However, when the application is restarted, Test_B may be retrieved.
Nikolaj Ahlberg-Pedersen
LPG: Hi! I was wondering what is meant by "application is restarted". Is this during development on a simulator perhaps?
Our SDK will connect initially to our servers with a user_id and a count of sessions played. If the session count is 0, then it's a new install and is included in a test potentially.
If your device for some reason is wiping the internal storage it will report session number 0 and be assigned again. This should not happen on normal use. For Android the session count is placed in more persistent storage I believe and might stick on simulator restarts. You should be able to see this using verbose logs perhaps .. and look for the event field "session_num". If it's 0 then SDK believes it's the first time it was started on the device.
We are working on making the integration easier to understand and verify/troubleshoot.
Let us know if you have more info.. thanks for reporting.