From felix.wolter at uni-mainz.de Mon Jul 27 13:48:46 2015 From: felix.wolter at uni-mainz.de (Felix Wolter) Date: Mon, 27 Jul 2015 13:48:46 +0200 Subject: [R list package] p value in test for design effects Message-ID: <55B61A9E.1000100@uni-mainz.de> Hello everybody, we have encountered some problems using the test for design effects in the list package. The outcome is pasted below. We would be very happy if someone could help us with the interpretation of the p-value of 1.227, which is greater than 1. Does this make sense? Or is the data unsuited for this test - as you can see, the distribution is heavily skewed with many respondents answering "0" (this is due to a special design of our ICT procedure). Thank you very much in advance! And kind greetings Felix > test.value.aushei <- ict.test(aushei$y, aushei$pct, J = 3, alpha=0.05, gms = TRUE, pi.table = TRUE) > print(test.value.aushei) Test for List Experiment Design Effects Estimated population proportions est. s.e. pi(y = 0, t = 1) 0.1178 0.0418 pi(y = 1, t = 1) 0.0356 0.0245 pi(y = 2, t = 1) 0.0067 0.0104 pi(y = 3, t = 1) 0.0045 0.0045 pi(y = 0, t = 0) 0.7364 0.0297 pi(y = 1, t = 0) 0.0686 0.0345 pi(y = 2, t = 0) 0.0280 0.0184 pi(y = 3, t = 0) 0.0024 0.0083 Bonferroni-corrected p-value If this value is below alpha, you reject the null hypothesis of no design effect. If it is above alpha, you fail to reject the null. Sensitive Item 1 1.227221 -- Dr. Felix Wolter Institut f?r Soziologie Johannes Gutenberg-Universit?t Mainz Georg-Forster-Geb?ude, Raum 03-446 Jakob-Welder-Weg 12 D-55128 Mainz Tel: 06131 - 39 20831 Fax: 06131 - 39 26157 Mail: felix.wolter at uni-mainz.de From kimai at Princeton.Edu Mon Jul 27 13:58:46 2015 From: kimai at Princeton.Edu (Kosuke Imai) Date: Mon, 27 Jul 2015 07:58:46 -0400 Subject: [R list package] p value in test for design effects In-Reply-To: <55B61A9E.1000100@uni-mainz.de> References: <55B61A9E.1000100@uni-mainz.de> Message-ID: Hi Felix, It's a typo in the output: the pvalue should be max(bonferroni.p-value, 1). So, your p-value is 1. This can be seen from the fact that your estimate is never negative. The test based on design effects flags when these estimates of proportions become negative (proportions should always be non-negative). The question is when that occurs whether or not a negative value is within sampling error. Hope this makes sense. We will fix the package soon. Best, Kosuke --------------------------------------------------------- Kosuke Imai Office: Corwin Hall 036 Professor Phone: 609-258-6601 Department of Politics Fax: 609-258-1110 Princeton University Email: kimai at Princeton.Edu Princeton, NJ 08544-1012 http://imai.princeton.edu --------------------------------------------------------- On Mon, Jul 27, 2015 at 7:48 AM, Felix Wolter wrote: > Hello everybody, > > we have encountered some problems using the test for design effects in the > list package. The outcome is pasted below. We would be very happy if > someone could help us with the interpretation of the p-value of 1.227, > which is greater than 1. Does this make sense? Or is the data unsuited for > this test - as you can see, the distribution is heavily skewed with many > respondents answering "0" (this is due to a special design of our ICT > procedure). > > Thank you very much in advance! > And kind greetings > > Felix > > > > test.value.aushei <- ict.test(aushei$y, aushei$pct, J = 3, alpha=0.05, > gms = TRUE, pi.table = TRUE) > > print(test.value.aushei) > > Test for List Experiment Design Effects > > Estimated population proportions > est. s.e. > pi(y = 0, t = 1) 0.1178 0.0418 > pi(y = 1, t = 1) 0.0356 0.0245 > pi(y = 2, t = 1) 0.0067 0.0104 > pi(y = 3, t = 1) 0.0045 0.0045 > pi(y = 0, t = 0) 0.7364 0.0297 > pi(y = 1, t = 0) 0.0686 0.0345 > pi(y = 2, t = 0) 0.0280 0.0184 > pi(y = 3, t = 0) 0.0024 0.0083 > > Bonferroni-corrected p-value > If this value is below alpha, you reject the null hypothesis of no design > effect. If it is above alpha, you fail to reject the null. > > Sensitive Item 1 > 1.227221 > > > > -- > Dr. Felix Wolter > Institut f?r Soziologie > Johannes Gutenberg-Universit?t Mainz > Georg-Forster-Geb?ude, Raum 03-446 > Jakob-Welder-Weg 12 > D-55128 Mainz > > Tel: 06131 - 39 20831 > Fax: 06131 - 39 26157 > Mail: felix.wolter at uni-mainz.de > _______________________________________________ > listpackage-discuss mailing list > listpackage-discuss at lists.r-forge.r-project.org > > https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/listpackage-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kimai at Princeton.Edu Mon Jul 27 19:48:08 2015 From: kimai at Princeton.Edu (Kosuke Imai) Date: Mon, 27 Jul 2015 13:48:08 -0400 Subject: [R list package] p value in test for design effects In-Reply-To: References: <55B61A9E.1000100@uni-mainz.de> Message-ID: Sorry. I meant to say "min" rather than "max"! Kosuke Imai Department of Politics Princeton University http://imai.princeton.edu On Mon, Jul 27, 2015 at 7:58 AM, Kosuke Imai wrote: > Hi Felix, > > It's a typo in the output: the pvalue should be max(bonferroni.p-value, > 1). So, your p-value is 1. This can be seen from the fact that your > estimate is never negative. The test based on design effects flags when > these estimates of proportions become negative (proportions should always > be non-negative). The question is when that occurs whether or not a > negative value is within sampling error. Hope this makes sense. We will > fix the package soon. > > Best, > Kosuke > > --------------------------------------------------------- > Kosuke Imai Office: Corwin Hall 036 > Professor Phone: 609-258-6601 > Department of Politics Fax: 609-258-1110 > Princeton University Email: kimai at Princeton.Edu > Princeton, NJ 08544-1012 http://imai.princeton.edu > --------------------------------------------------------- > > On Mon, Jul 27, 2015 at 7:48 AM, Felix Wolter > wrote: > >> Hello everybody, >> >> we have encountered some problems using the test for design effects in >> the list package. The outcome is pasted below. We would be very happy if >> someone could help us with the interpretation of the p-value of 1.227, >> which is greater than 1. Does this make sense? Or is the data unsuited for >> this test - as you can see, the distribution is heavily skewed with many >> respondents answering "0" (this is due to a special design of our ICT >> procedure). >> >> Thank you very much in advance! >> And kind greetings >> >> Felix >> >> >> > test.value.aushei <- ict.test(aushei$y, aushei$pct, J = 3, alpha=0.05, >> gms = TRUE, pi.table = TRUE) >> > print(test.value.aushei) >> >> Test for List Experiment Design Effects >> >> Estimated population proportions >> est. s.e. >> pi(y = 0, t = 1) 0.1178 0.0418 >> pi(y = 1, t = 1) 0.0356 0.0245 >> pi(y = 2, t = 1) 0.0067 0.0104 >> pi(y = 3, t = 1) 0.0045 0.0045 >> pi(y = 0, t = 0) 0.7364 0.0297 >> pi(y = 1, t = 0) 0.0686 0.0345 >> pi(y = 2, t = 0) 0.0280 0.0184 >> pi(y = 3, t = 0) 0.0024 0.0083 >> >> Bonferroni-corrected p-value >> If this value is below alpha, you reject the null hypothesis of no design >> effect. If it is above alpha, you fail to reject the null. >> >> Sensitive Item 1 >> 1.227221 >> >> >> >> -- >> Dr. Felix Wolter >> Institut f?r Soziologie >> Johannes Gutenberg-Universit?t Mainz >> Georg-Forster-Geb?ude, Raum 03-446 >> Jakob-Welder-Weg 12 >> D-55128 Mainz >> >> Tel: 06131 - 39 20831 >> Fax: 06131 - 39 26157 >> Mail: felix.wolter at uni-mainz.de >> _______________________________________________ >> listpackage-discuss mailing list >> listpackage-discuss at lists.r-forge.r-project.org >> >> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/listpackage-discuss >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: