All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.

Related Documents

Share

Description

Supplementary material for Gene selection and classification of microarray data using random forest Ramón Díaz-Uriarte, Sara Alvarez de Andrés Bioinformatics Unit, Cytogenetics Unit Spanish National Cancer

Transcript

Supplementary material for Gene selection and classification of microarray data using random forest Ramón Díaz-Uriarte, Sara Alvarez de Andrés Bioinformatics Unit, Cytogenetics Unit Spanish National Cancer Center (CNIO) Melchor Fernández Almagro 3 Madrid, Spain. 1 1 Variable importance from random forest Random forest returns several measures of variable importance. The most reliable measure of variable importance is based on the decrease of classification accuracy when values of a variable in a node of a tree are permuted randomly (Breiman, 2001; Bureau et al., 2003; Remlinger, 2004). This measure is sometimes reported as such, and sometimes it is reported after scaling it, or dividing by a quantity somewhat analogous to its standard error ( somewhat analogous because the data used to obtain that standard error are not truly independent, and thus the true standard error can be severely underestimated). We use in this paper the unscaled importance measure, because it allows us to compare directly runs with different settings of ntree and mtry (in contrast, scaled importances increase monotonically as we increase the value of ntree). 2 Microarray data sets The data sets Colon, Prostate, Lymphoma, SRBCT and Brain were obtained, as binary R files, from Marcel Dettling s web site The data sets and their preprocessing are fully described in Dettling & Bühlmann (2002). Leukemia dataset From Golub et al. (1999). The original data, from an Affymetrix chip, comprises 6817 genes, but after filtering as done by the authors we are left with 3051 genes. Filtering and preprocessing is described in the original paper and in Dudoit et al. (2002). We used the training data set of 38 cases (27 ALL and 11 AML) in the original paper (the observations in the test set are from a different lab and were collected at different times). This data set is available from [http://www-genome.wi.mit.edu/cgi-bin/cancer/datasets.cgi] and also from the Bioconductor package multtest ([http://www.bioconductor.org]). Adenocarcinoma dataset From Ramaswamy et al. (2003). We used the data from the 12 metastatic tumors and 64 primary tumors. The original data set included genes from Affymetrix chips. The data (DatasetA Tum vsmet.res), downloaded from [http://www-genome.wi.mit.edu/ cgi-bin/cancer/], had already been rescaled by the authors. We took the subset of 9376 genes according to the UniGene mapping, thresholded the data, and filtered by variation as explained by the authors. The final data set contains 9868 clones (several genes were represented by more than one clone); of these, 196 had constant values over all individuals. NCI 60 dataset From Ross et al. (2000). The data, from cdna arrays, can be obtained from [http:// genome-www.stanford.edu/sutech/download/nci60/index.html]. The raw data we used, which is the same as the data used in Dettling & Bühlmann (2003); Dudoit et al. (2002), is the one in the file figure3.cdt. As in Dettling & Bühlmann (2003); Dudoit et al. (2002) we filtered out genes with more than two missing observations and we also eliminated, because of small sample size, the two prostate cell line observations and the unknown observation. After filtering, we were left with a 61 x 5244 matrix, corresponding to eight different tumor types (note that, as done by previous authors, we did not average the two observations with triplicate hybridizations). As in Dudoit et al. (2002) we used 5-nearest neighbor imputation of missing data using the program GEPAS (Herrero et al., 2003) (http://gepas.bioinfo.cnio.es/cgi-bin/preprocess); unlike Dudoit et al. (2002), however, we measured gene similarity using Euclidean distance from the genes with complete data, instead of correlation: Troyanskaya et al. (2001) found Euclidean distance to be an appropriate metric. Finally, as in (Dudoit et al., 2002, p. 82) gene expression data were standardized so that arrays had mean 0 and variance 1 across variables (genes). Breast cancer dataset From van t Veer et al. (2002). The data were downloaded from [http:// (we used the files ArrayData less than 5yr.zip, ArrayData greater than 5yr.zip, ArrayData BRCA1.zip, corresponding to 34 patients that developed distant metastases within 5 years, 44 that remained disease-free for over 5 years, and 18 with BRCA1 germline mutations and 2 with BRCA2 mutations). As did by the authors, we selected only the genes that were significantly regulated (see their definition in the paper and supplementary material), which resulted in a total of 4869 clones. Because of the small sample size, we excluded the 2 patients with the BRCA2 mutation. We used 5-nearest neighbor imputation for the missing data, as for the NCI 60 data set. Finally, we excluded from the analyses the 10th subject from the set that developed metastases in less than 5 years (sample 54, IRI , in the original data files), because it had missing values out of the original clones, and was an outstanding outlying point both before and after imputation. The breast cancer dataset was used both for two 2 class comparison (those that developed metastases within 5 years vs. those that remain metastases free after 5 years) and for three group comparisons. 3 Generation of simulated data We have simulated data under different number of classes of patients (2, 3, 4), number of independent dimensions (1 to 3), and number of genes per dimension (5, 20, 100). In all cases, the number of subjects per class has been set to 25 (a number which is similar to, or smaller than, that of many microarray studies). The data have been simulated from a multivariate normal distribution. All genes have a variance of 1, and the correlation between genes within a dimension is 0.9, whereas the correlation between genes among dimensions is 0. In other words, the variance-covariance matrix is a block-diagonal matrix as: a a Σ = , a where a = The class means have been set so that the unconditional prediction error rate (see McLachlan (1992)) of a DLDA using one gene from each dimension is approximately 5%; and each dimension has the same relevance in separation. Specifically, the class means used are: One dimension: Two classes: µ 1 = 1.65, µ 2 = Three classes: µ 1 = 3.58, µ 2 = 0, µ 3 = Four classes: µ 1 = 3.7, µ 2 = 0, µ 3 = 3.7, µ 4 = 7.4. Two dimensions: Two classes:µ 1 = [ 1.18, 1.18], µ 2 = [1.18, 1.18].... Three classes: µ 1 = [0, 0], µ 2 = [3.88 cos(15), 3.88 sin(15)], µ 3 = [3.88 cos(75), 3.88 sin(75)]. Four classes: µ 1 = [1, 1], µ 2 = [4.95, 1], µ 3 = [1, 4.95], µ 4 = [4.95, 4.95]. Three dimensions: Two classes:µ 1 = [ 0.98, 0.98, 0.98], µ 2 = [0.98, 0.98, 0.98]. Three classes: µ 1 = [2.76, 0, 0], µ 2 = [0, 2.76, 0], µ 3 = [0, 0, 2.76]. Four classes: µ 1 = [2.96, 0, 0], µ 2 = [0, 2.96, 0], µ 3 = [0, 0, 2.96], µ 4 = [2.96, 2.96, 2.96] After the genes that belong to the dimensions are generated, we add another 2000 N (0, 1) variables and another 2000 U[ 1, 1] variables to the matrix of genes. For each combination of number of dimensions * number of classes * number of genes per dimension we generate 4 data sets. 4 Choosing mtry and ntree Figure error.vs.mtry.pdf shows the OOB error rate plotted against the mtry factor for different ntree and nodesize. The mtry factor = {0, 0.05, 0.1, 0.17, 0.25, 0.33, 0.5, 0.75, 0.8, 1, 1.15, 1.33, 1.5, 2, 3, 4, 5, 6, 8, 10, 13}, where an mtry factor of 0 means mtry = 1 variable. Values of ntree = 1000, 2000, 5000, 10000, for the simulated data (both with and without signal) and ntree = 1000, 2000, 5000, 10000, 20000, for the real microarray data sets. The values of nodesize were 1 (the default) and 5. 3 5 Backwards elimination of variables using OOB error 5.1 Simulated data Classes # Vars Error rate 94 (10, 24, 75) (15, 38, 94) (12, 30, 75) (12, 38, 94) (19, 48, 94) (24, 48, 75) (24, 48, 75) (19, 48, 94) (28, 60, 94) (30, 48, 94) (30, 60, 94) (38, 60, 94) Table 1: simplify.no.signal.02 Number of variables selected and error rate (estimated using the.632+ bootstrap method, with 200 bootstrap samples) from simulated data without signal. Results shown for four replicates of each condition. Values in parenthesis are the 25th percentile, median, and 75th percentile of the number of variables selected when running the procedure on the bootstrap samples. The parameters used where f raction.dropped = 0.2, ntree = 2000, ntreeiterat = 1000, nodesize = 1, mtryf actor = 1, serule = 1. The error rates estimated are comparable to the error rates from always betting on the most common class (in this case all are equiprobable, and those error rates correspond to 50% in the 2 class case, 66% in the 3 class case and 75% in the 4 class case). Classes # Vars Error rate 65 (5, 33, 65) (9, 33, 131) (17, 33, 131) (17, 65, 131) (33, 65, 131) (33, 49, 131) (33, 65, 131) (33, 65, 131) (33, 65, 131) (33, 65, 131) (33, 65, 131) (33, 65, 131) Table 2: simplify.no.signal.05 Number of variables selected and error rate (estimated using the.632+ bootstrap method, with 200 bootstrap samples) from simulated data without signal. Results shown for four replicates of each condition. Values in parenthesis are the 25th percentile, median, and 75th percentile of the number of variables selected when running the procedure on the bootstrap samples. The parameters used where fraction.dropped = 0.5, ntree = 5000, nodesize = 1, mtryf actor = 1, serule = 1. Recall that, with f raction.dropped = 0.5, we eliminate 50% of the variables at each iteration (see text), which explains the set of values 33, 65, 131, 263 (number of variables in step t = number of variables in step t round (number of variables in step t + 1 * 0.5)). The error rates estimated are comparable to the error rates from always betting on the most common class (in this case all are equiprobable, and those error rates correspond to 50% in the 2 class case, 66% in the 3 class case and 75% in the 4 class case). 4 Classes Dimensions Genes/dimension # Vars Error rate (2, 2, 3) (2, 2, 3) (2, 2, 3) (2, 3, 96) (2, 2, 5) (2, 2, 3) (3, 17, 188) (2, 5, 178) (2, 2, 3) (2, 2, 3) (2, 3, 5) (2, 2, 3) (3, 5, 176) (3, 5, 10) (3, 4, 8) (6, 42, 276) (3, 8, 19) (3, 5, 9) (2, 7, 220) (2, 2, 4) (2, 2, 3) (2, 4, 8) (2, 3, 5) (4, 8, 12) (12, 114, 276) (4, 8, 24) (3, 7, 17) The 25th percentile, median, and 75th percentile of number of variables selected when running on another 10 data sets (generated with the same parameters) were: 8.25, , , 38, , 2.00, , , , 3, , 6.00, , 52.0, , 46.0, , 15.00, , 4.50, Table 3: simplify.signal.02 Number of variables selected and error rate (estimated using the.632+ bootstrap method, with 200 bootstrap samples) from simulated data with signal. Values in parenthesis are the 25th percentile, median, and 75th percentile of the number of variables selected when running the procedure on the bootstrap samples. The parameters used where f raction.dropped = 0.2, ntree = 2000, ntreeiterat = 1000, nodesize = 1, mtryf actor = 1, serule = 1 5 Classes Dimensions Genes/dimension # Vars Error rate (2, 2, 2) (2, 2, 3) (2, 2, 3) (2, 2, 7) (2, 2, 3) (2, 2, 3) (2, 7, 125) (2, 3, 253) (2, 2, 3) (2, 2, 3) (2, 2, 3) (2, 2, 3) (3, 7, 15) (3, 7, 15) (2, 5, 9) (7, 47, 251) (3, 11, 31) (3, 5, 9) (2, 3, 63) (2, 2, 3) (2, 2, 2) (2, 7, 15) (2, 3, 7) (9, 17, 33) (15, 63, 251) (3, 7, 31) (3, 9, 33) The 25th percentile, median, and 75th percentile of number of variables selected when running on another 10 data sets (generated with the same parameters) were: 15, 47, , 2, , 5, , 47, 251. Table 4: simplify.signal.05 Number of variables selected and error rate (estimated using the.632+ bootstrap method, with 200 bootstrap samples) from simulated data with signal. Values in parenthesis are the 25th percentile, median, and 75th percentile of the number of variables selected when running the procedure on the bootstrap samples. The parameters used where f raction.dropped = 0.5, ntree = 5000, ntreeiterat = 5000, nodesize = 1, mtryf actor = 1, serule = 1 6 5.2 Real microarray data sets 7 Data set Error rate # Vars # Vars bootstrap Freq. vars mtry factor = 1, s.e. = 0, ntree = 5000 Leukemia (2, 2) 0.42 (0.32, 0.5) 1 Breast 2 cl (3, 19) 0.14 (0.1, 0.19) Breast 3 cl (9, 39) 0.2 (0.14, 0.31) NCI (41, 81) 0.1 (0.05, 0.18) Adenocar (2, 9) 0.06 (0.06, 0.2) Brain (11, 43) 0.29 (0.26, 0.4) Colon (3, 15) 0.3 (0.25, 0.4) Lymphoma (6, 63) 0.36 (0.29, 0.47) Prostate (2, 23) 0.41 (0.28, 0.5) Srbct (19, 37) 0.76 (0.56, 0.92) mtry factor = 13, s.e. = 0, ntree = 5000 Leukemia (2, 2) 0.4 (0.29, 0.52) 1 Breast 2 cl (4, 39) 0.14 (0.09, 0.22) Breast 3 cl (5, 39) 0.12 (0.09, 0.16) NCI (41, 81) 0.28 (0.2, 0.4) Adenocar (2, 9) 0.14 (0.12, 0.16) Brain (11, 43) 0.28 (0.22, 0.46) Colon (2, 15) 0.45 (0.38, 0.49) Lymphoma (15, 125) 0.46 (0.35, 0.57) Prostate (5, 755) 0.18 (0.12, 0.27) Srbct (37, 73) 0.84 (0.66, 0.98) mtry factor = 1, s.e. = 1, ntree = 5000 Leukemia (2, 2) 0.37 (0.29, 0.46) 1 Breast 2 cl (2, 6) 0.08 (0.05, 0.13) Breast 3 cl (3, 19) 0.31 (0.27, 0.34) NCI (21, 81) 0.34 (0.19, 0.42) Adenocar (2, 3) 0.17 (0.16, 0.18) 1 Brain (11, 43) 0.34 (0.28, 0.48) Colon (2, 3) 0.29 (0.19, 0.3) Lymphoma (7, 125) 0.31 (0.24, 0.46) Prostate (2, 11) 0.9 (0.8, 1) 1 Srbct (19, 37) 0.36 (0.2, 0.58) mtry factor = 13, s.e. = 1, ntree = 5000 Leukemia (2, 2) 0.45 (0.32, 0.6) 1 Breast 2 cl (2, 9) 0.28 (0.16, 0.32) Breast 3 cl (3, 9) 0.08 (0.05, 0.15) NCI (21, 41) 0.13 (0.08, 0.24) Adenocar (2, 3) 0.14 (0.08, 0.14) Brain (21, 43) 0.33 (0.27, 0.58) Colon (2, 7) 0.31 (0.25, 0.34) Lymphoma (7, 125) 0.38 (0.29, 0.5) Prostate (2, 23) 0.96 (0.92, 1) 1 Srbct (37, 73) 0.8 (0.68, 0.97) 1 Since there are only two variables, the values here are the actual frequencies of those two variables, not the 25th and 75th percentiles. Table 5: stability-5000 Error rate and stability of results of backwards elimination of variables using OOB error, evaluated using 200 bootstrap samples. Results for f raction.dropped = 0.5, ntree = 5000, ntreeiterat = Error rate is the error rate estimated using bootstrap method. # Vars denotes the number of variables selected on the original data set. # Vars bootstrap shows the median (1st quartile, 3rd quartile) number of variables selected when the procedure is run on the bootstrap samples. Freq. vars is the median (1st quartile, 3rd quartile) of the frequency with which each variable in the original data set appears in the variables selected when the procedure is run on the bootstrap samples. 8 Data set Error rate # Vars # Vars bootstrap Freq. vars mtry factor = 1, s.e. = 0, ntree = Leukemia (2, 2) 0.44 (0.28, 0.61) 1 Breast 2 cl (3, 19) 0.18 (0.14, 0.24) Breast 3 cl (9, 39) 0.21 (0.14, 0.32) NCI (41, 81) 0.12 (0.07, 0.19) Adenocar (2, 9) 0.12 (0.1, 0.19) Brain (11, 43) 0.34 (0.25, 0.54) Colon (2, 7) 0.23 (0.18, 0.34) Lymphoma (7, 125) 0.25 (0.18, 0.37) Prostate (3, 23) 0.28 (0.23, 0.48) Srbct (19, 37) 0.3 (0.16, 0.52) mtry factor = 13, s.e. = 0, ntree = Leukemia (2, 2) 0.4 (0.22, 0.57) 1 Breast 2 cl (3, 39) 0.21 (0.15, 0.29) Breast 3 cl (9, 39) 0.12 (0.09, 0.18) NCI (41, 81) 0.28 (0.22, 0.42) Adenocar (2, 9) 0.22 (0.2, 0.24) Brain (21, 43) 0.4 (0.34, 0.63) Colon (2, 15) 0.26 (0.23, 0.35) Lymphoma (7, 125) 0.29 (0.13, 0.44) Prostate (3, 755) 0.19 (0.14, 0.26) Srbct (37, 73) 0.62 (0.5, 0.84) mtry factor = 1, s.e. = 1, ntree = Leukemia (2, 2) 0.47 (0.32, 0.61) 1 Breast 2 cl (2, 9) 0.16 (0.1, 0.26) Breast 3 cl (3, 19) 0.16 (0.12, 0.24) NCI (21, 81) 0.38 (0.26, 0.48) Adenocar (2, 3) 0.08 (0.07, 0.13) Brain (11, 43) 0.32 (0.25, 0.56) Colon (2, 3) 0.28 (0.22, 0.36) Lymphoma (3, 125) 0.41 (0.33, 0.47) Prostate (2, 11) 0.94 (0.89, 1) 1 Srbct (19, 37) 0.32 (0.17, 0.54) mtry factor = 13, s.e. = 1, ntree = Leukemia (2, 2) 0.43 (0.26, 0.61) 1 Breast 2 cl (2, 9) 0.13 (0.1, 0.18) Breast 3 cl (3, 9) 0.15 (0.08, 0.2) NCI (21, 41) 0.17 (0.11, 0.28) Adenocar (2, 5) 0.12 (0.1, 0.16) Brain (21, 43) 0.46 (0.37, 0.64) Colon (2, 7) 0.34 (0.31, 0.36) Lymphoma (7, 125) 0.36 (0.29, 0.48) Prostate (2, 47) 0.95 (0.91, 1) 1 Srbct (19, 73) 0.82 (0.63, 0.94) 1 Since there are only two variables, the values here are the actual frequencies of those two variables, not the 25th and 75th percentiles. Table 6: stability Error rate and stability of results of backwards elimination of variables using OOB error, evaluated using 200 bootstrap samples. Results for f raction.dropped = 0.5, ntree = 20000, ntreeiterat = Error rate is the error rate estimated using bootstrap method. # Vars denotes the number of variables selected on the original data set. # Vars bootstrap shows the median (1st quartile, 3rd quartile) number of variables selected when the procedure is run on the bootstrap samples. Freq. vars is the median (1st quartile, 3rd quartile) of the frequency with which each variable in the original data set appears in the variables selected when the procedure is run on the bootstrap samples. 9 Data set Error rate # Vars # Vars bootstrap Freq. vars mtry factor = 1, s.e. = 0, ntree = 5000, ntreeiterat = 2000 Leukemia (2, 2) 0.4 (0.3, 0.5) 1 Breast 2 cl (5, 23) 0.2 (0.14, 0.29) Breast 3 cl (9, 31) 0.14 (0.1, 0.22) NCI (30, 94) 0.12 (0.08, 0.21) Adenocar (2, 8) 0.14 (0.13, 0.18) Brain (8, 22) 0.24 (0.17, 0.44) Colon (2, 9) 0.19 (0.18, 0.29) Lymphoma (5, 73) 0.34 (0.24, 0.42) Prostate (2, 12) 0.21 (0.16, 0.38) Srbct (11, 27) 0.54 (0.36, 0.88) mtry factor = 1, s.e. = 1, ntree = 5000, ntreeiterat = 2000 Leukemia (2, 2) 0.38 (0.26, 0.52) 1 Breast 2 cl (3, 7) 0.22 (0.1, 0.26) Breast 3 cl (4, 14) 0.2 (0.1, 0.3) NCI (19, 60) 0.32 (0.29, 0.44) Adenocar (2, 4) 0.08 (0.06, 0.09) Brain (7, 22) 0.28 (0.22, 0.46) Colon (2, 5) 0.3 (0.24, 0.38) Lymphoma (4, 91) 0.3 (0.21, 0.4) Prostate (2, 5) 0.93 (0.86, 1) 1 Srbct (11, 27) 0.27 (0.18, 0.45) mtry factor = 1, s.e. = 0, ntree = 2000, ntreeiterat = 1000 Leukemia (2, 2) 0.38 (0.29, 0.48) 1 Breast 2 cl (5, 23) 0.15 (0.1, 0.28) Breast 3 cl (9, 31) 0.08 (0.04, 0.13) NCI (30, 94) 0.1 (0.06, 0.19) Adenocar (2, 8) 0.14 (0.12, 0.15) Brain (7, 22) 0.18 (0.09, 0.25) Colon (3, 12) 0.29 (0.19, 0.42) Lymphoma (4, 58) 0.26 (0.18, 0.38) Prostate (3, 14) 0.22 (0.17, 0.43) Srbct (11, 27) 0.1 (0.04, 0.29) mtry factor = 1, s.e. = 1, ntree = 2000, ntreeiterat = 1000 Leukemia (2, 2) 0.4 (0.32, 0.5) 1 Breast 2 cl (2, 7) 0.12 (0.07, 0.17) Breast 3 cl (4, 14) 0.27 (0.22, 0.31) NCI (19, 60) 0.26 (0.17, 0.38) Adenocar (2, 5) 0.06 (0.03, 0.12) Brain (7, 22) 0.26 (0.14, 0.46) Colon (2, 6) 0.36 (0.32, 0.36) Lymphoma (5, 73) 0.32 (0.24, 0.42) Prostate (2, 5) 0.9 (0.82, 0.99) 1 Srbct (11, 34) 0.57 (0.4, 0.88) 1 Since there are only two variables, the values here are the actual frequencies of those two variables, not the 25th and 75th percentiles. Table 7: stability-02 Error rate and stability of results of backwards elimination of variables using OOB error, evaluated using 200 bootstrap samples. Results for f raction.dropped = 0.2. Error rate is the error rate estimated using bootstrap method. # Vars denotes the number of variables selected on the original data set. # Vars bootstrap shows the median (1st quartile, 3rd quartile) number of variables selected when the procedure is run on the bootstrap samples. Freq. vars is the median (1st quartile, 3rd quartile) of the frequency with which each variable in the original data set appears in the variables selected when the procedure is run on the bootstrap samples. 10 Data set Error rate # Vars # Vars bootstrap Freq. vars Shrunken centroids; mimimizing error rate then maximizing log-likelihood Leukemia (102, 3051) 0.64 (0.6, 0.68) Breast 2 cl (33, 340) 0.58 (0.54, 0.65) Breast 3 cl (3272, 4869) 0.9 (0.86, 0.92) NCI (3485, 5232) 0.82 (0.72, 0.92) Adenocar (4, 20) 0.66 (0.66, 0.66) 1 Brain (459, 4026) 0.42 (0.32, 0.55) Colon (20, 70) 0.77 (0.57, 0.89) Lymphoma (2664, 4026) 0.88 (0.82, 0.92) Prostate (4, 14) 0.57 (0.37, 0.78) Srbct (130, 470) 0.68 (0.56, 0.86) Shrunken centroids; mimimizing error rate then minimizing number of genes selected Leukemia (14, 504) 0.48 (0.45, 0.59) Breast 2 cl (24, 296) 0.54 (0.51, 0.66) Breast 3 cl (2379, 4804) 0.84 (0.78, 0

Search

Similar documents

Related Search

Centers For Disease Control And PreventionMinute And Second Of ArcStates And Territories Of IndiaInternational Statistical Classification Of DProvinces And Territories Of CanadaChapters And Verses Of The BibleStates And Territories Of AustraliaThe History Of The Decline And Fall Of The RoBattles And Operations Of World War IiClassification Of Minerals

We Need Your Support

Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks