我的weka github地址Q?a >https://github.com/changedi/wekaQ只读git路径Qgit://github.com/changedi/weka.gitQ欢qfork
]]>
http://weka.wikispaces.com/ARFF+files+from+Text+Collections
weka可以以目录Ş式读入数?/strong>?br />然后再简单说一下weka在做文本特征内容处理时候需要注意的东西Q?br />声明一点，在weka的gui下是没法使用q个功能的：(x)以目录Ş式读入数据?br />首先Q把要处理的数据写入到这L(fng)目录l构下：(x)
... | +- text_example | +- class1 | | | + file1.txt | | | + file2.txt | | | ... | +- class2 | | | + another_file1.txt | | | + another_file2.txt | | | ...
然后在源码包下，命o(h)行执?java weka.core.converters.TextDirectoryLoader -dir text_example > text_example.arff
其中text_example是数据所在的目录Q而后面的arff文g是生成的arff文g。另外值得补充的一Ҏ(gu)在获得这L(fng)arff后哦Q文本内Ҏ(gu)作ؓ(f)一个字W串特征存在的，也就是说生成的arff是一个特征项加一个类标签Q其中的cL是text_example目录下classX子目录的名字。ؓ(f)?jin)更方便使用Qweka提供?jin)一个有监督的属性过滤器Q帮助分词（q里指英文的splitQ?——StringToWordVectorQ这个是可以做TF/IDF的~~~
下面的简单代码可以完成一个分c：(x)
1
import weka.core.*;
2import weka.core.converters.*;
3import weka.classifiers.trees.*;
4import weka.filters.*;
5import weka.filters.unsupervised.attribute.*;
6
7import java.io.*;
8
9
/**
10
* Example class that converts HTML files stored in a directory structure into
11 * and ARFF file using the TextDirectoryLoader converter. It then applies the
12 * StringToWordVector to the data and feeds a J48 classifier with it.
13 *
14 * @author FracPete (fracpete at waikato dot ac dot nz)
15
*/
16public class TextCategorizationTest {
17
18
/**
19 * Expects the first parameter to point to the directory with the text files.
20 * In that directory, each sub-directory represents a class and the text
21 * files in these sub-directories will be labeled as such.
22 *
23 * @param args the commandline arguments
24 * @throws Exception if something goes wrong
25
*/
26 public static void main(String[] args) throws Exception {
27 // convert the directory into a dataset
28 TextDirectoryLoader loader = new TextDirectoryLoader();
29 loader.setDirectory(new File("./text_example"));
30 Instances dataRaw = loader.getDataSet();
31 System.out.println("\n\nImported data:\n\n" + dataRaw.numClasses());
32
33 // apply the StringToWordVector
34 // (see the source code of setOptions(String[]) method of the filter
35 // if you want to know which command-line option corresponds to which
36 // bean property)
37 StringToWordVector filter = new StringToWordVector();
38 filter.setInputFormat(dataRaw);
39 Instances dataFiltered = Filter.useFilter(dataRaw, filter);
40 System.out.println("\n\nFiltered data:\n\n" + dataFiltered);
41
42 // train J48 and output model
43 J48 classifier = new J48();
44 classifier.buildClassifier(dataFiltered);
45 System.out.println("\n\nClassifier model:\n\n" + classifier);
46 }
47}
48
import weka.core.*;2
import weka.core.converters.*;3
import weka.classifiers.trees.*;4
import weka.filters.*;5
import weka.filters.unsupervised.attribute.*;6
7
import java.io.*;8
9
/**
10
* Example class that converts HTML files stored in a directory structure into 11
* and ARFF file using the TextDirectoryLoader converter. It then applies the12
* StringToWordVector to the data and feeds a J48 classifier with it.13
*14
* @author FracPete (fracpete at waikato dot ac dot nz)15
*/16
public class TextCategorizationTest {
17
18
/**
19
* Expects the first parameter to point to the directory with the text files.20
* In that directory, each sub-directory represents a class and the text21
* files in these sub-directories will be labeled as such.22
*23
* @param args the commandline arguments24
* @throws Exception if something goes wrong25
*/26
public static void main(String[] args) throws Exception {27
// convert the directory into a dataset28
TextDirectoryLoader loader = new TextDirectoryLoader();29
loader.setDirectory(new File("./text_example"));30
Instances dataRaw = loader.getDataSet();31
System.out.println("\n\nImported data:\n\n" + dataRaw.numClasses());32
33
// apply the StringToWordVector34
// (see the source code of setOptions(String[]) method of the filter35
// if you want to know which command-line option corresponds to which36
// bean property)37
StringToWordVector filter = new StringToWordVector();38
filter.setInputFormat(dataRaw);39
Instances dataFiltered = Filter.useFilter(dataRaw, filter);40
System.out.println("\n\nFiltered data:\n\n" + dataFiltered);41
42
// train J48 and output model43
J48 classifier = new J48();44
classifier.buildClassifier(dataFiltered);45
System.out.println("\n\nClassifier model:\n\n" + classifier);46
}47
}48
最后，我还是徏议数据徏模和生成都自己写E序Q数据准备往往自己的程序才能准的控制Qweka最多是帮我们做一下selection和classification?br />另外补充一点，很多朋友问到?jin)如何做文本分类Q好吧，如果大家懒得去读paper的话Q首先我普及(qing)一点，不管什么分c，分类器基本是可以通用的，注意是基本。关键是模型的构建和特征的生成。至于文本分cM用到的特征，TF*IDFq有其他如互信息Q卡方统计，期望交叉늭{，公式摆在那里Q计真的不难。因为就我接触过的分c问题，文本分类的特征计应该是很容易的?jin)?br />
]]>
1/**
2 *
3 */
4package edu.tju.ikse.mi.util;
5
6import java.io.File;
7import java.io.IOException;
8import java.util.Random;
9
10import weka.attributeSelection.ASEvaluation;
11import weka.attributeSelection.ASSearch;
12import weka.attributeSelection.AttributeSelection;
13import weka.attributeSelection.BestFirst;
14import weka.attributeSelection.CfsSubsetEval;
15import weka.core.Instances;
16import weka.core.converters.ArffLoader;
17
18/**
19 * @author Jia Yu
20 * @date 2010-11-23
21 */
22public class WekaSelector {
23
24 private ArffLoader loader;
25 private Instances dataSet;
26 private File arffFile;
27 private int sizeOfDataset;
28 private int numOfOldAttributes;
29 private int numOfNewAttributes;
30 private int classIndex;
31 private int[] selectedAttributes;
32
33 public WekaSelector(File file) throws IOException {
34 loader = new ArffLoader();
35 arffFile = file;
36 loader.setFile(arffFile);
37 dataSet = loader.getDataSet();
38 sizeOfDataset = dataSet.numInstances();
39 numOfOldAttributes = dataSet.numAttributes();
40 classIndex = numOfOldAttributes - 1;
41 dataSet.setClassIndex(classIndex);
42 }
43
44 public void select() throws Exception {
45 ASEvaluation evaluator = new CfsSubsetEval();
46 ASSearch search = new BestFirst();
47 AttributeSelection eval = null;
48
49 eval = new AttributeSelection();
50 eval.setEvaluator(evaluator);
51 eval.setSearch(search);
52
53 eval.SelectAttributes(dataSet);
54 numOfNewAttributes = eval.numberAttributesSelected();
55 selectedAttributes = eval.selectedAttributes();
56 System.out.println("result is "+eval.toResultsString());
57 /*
58 Random random = new Random(seed);
59 dataSet.randomize(random);
60 if (dataSet.attribute(classIndex).isNominal()) {
61 dataSet.stratify(numFolds);
62 }
63 for (int fold = 0; fold < numFolds; fold++) {
64 Instances train = dataSet.trainCV(numFolds, fold, random);
65 eval.selectAttributesCVSplit(train);
66 }
67 System.out.println("result is "+eval.CVResultsString());
68 */
69 System.out.println("old number of Attributes is "+numOfOldAttributes);
70 System.out.println("new number of Attributes is "+numOfNewAttributes);
71 for(int i=0;i<selectedAttributes.length;i++){
72 System.out.println(selectedAttributes[i]);
73 }
74 }
75
76 /**
77 * @param args
78 */
79 public static void main(String[] args) {
80 // TODO Auto-generated method stub
81 File file = new File("iris.arff");
82 try {
83 WekaSelector ws = new WekaSelector(file);
84 ws.select();
85
86 } catch (IOException e) {
87 // TODO Auto-generated catch block
88 e.printStackTrace();
89 } catch (Exception e) {
90 // TODO Auto-generated catch block
91 e.printStackTrace();
92 }
93
94 }
95
96}
97
/**2
* 3
*/4
package edu.tju.ikse.mi.util;5
6
import java.io.File;7
import java.io.IOException;8
import java.util.Random;9
10
import weka.attributeSelection.ASEvaluation;11
import weka.attributeSelection.ASSearch;12
import weka.attributeSelection.AttributeSelection;13
import weka.attributeSelection.BestFirst;14
import weka.attributeSelection.CfsSubsetEval;15
import weka.core.Instances;16
import weka.core.converters.ArffLoader;17
18
/**19
* @author Jia Yu20
* @date 2010-11-2321
*/22
public class WekaSelector {23
24
private ArffLoader loader;25
private Instances dataSet;26
private File arffFile;27
private int sizeOfDataset;28
private int numOfOldAttributes;29
private int numOfNewAttributes;30
private int classIndex;31
private int[] selectedAttributes;32
33
public WekaSelector(File file) throws IOException {34
loader = new ArffLoader();35
arffFile = file;36
loader.setFile(arffFile);37
dataSet = loader.getDataSet();38
sizeOfDataset = dataSet.numInstances();39
numOfOldAttributes = dataSet.numAttributes();40
classIndex = numOfOldAttributes - 1;41
dataSet.setClassIndex(classIndex);42
}43
44
public void select() throws Exception {45
ASEvaluation evaluator = new CfsSubsetEval();46
ASSearch search = new BestFirst();47
AttributeSelection eval = null;48
49
eval = new AttributeSelection();50
eval.setEvaluator(evaluator);51
eval.setSearch(search);52
53
eval.SelectAttributes(dataSet);54
numOfNewAttributes = eval.numberAttributesSelected();55
selectedAttributes = eval.selectedAttributes();56
System.out.println("result is "+eval.toResultsString());57
/*58
Random random = new Random(seed);59
dataSet.randomize(random);60
if (dataSet.attribute(classIndex).isNominal()) {61
dataSet.stratify(numFolds);62
}63
for (int fold = 0; fold < numFolds; fold++) {64
Instances train = dataSet.trainCV(numFolds, fold, random);65
eval.selectAttributesCVSplit(train);66
}67
System.out.println("result is "+eval.CVResultsString());68
*/69
System.out.println("old number of Attributes is "+numOfOldAttributes);70
System.out.println("new number of Attributes is "+numOfNewAttributes);71
for(int i=0;i<selectedAttributes.length;i++){72
System.out.println(selectedAttributes[i]);73
}74
}75
76
/**77
* @param args78
*/79
public static void main(String[] args) {80
// TODO Auto-generated method stub81
File file = new File("iris.arff");82
try {83
WekaSelector ws = new WekaSelector(file);84
ws.select();85
86
} catch (IOException e) {87
// TODO Auto-generated catch block88
e.printStackTrace();89
} catch (Exception e) {90
// TODO Auto-generated catch block91
e.printStackTrace();92
}93
94
}95
96
}97
其中的注释部分是使用交叉验证的部分。默认是十折?sh)叉验证Q当然这个可以通过setҎ(gu)讄。具体的使用或者用到reduce dimensionality的方法大家可以参看源代码。毕竟weka开源很是方ѝ源代码涉及(qing)到的cM要是查看weka.attributeSelection.AttributeSelectioncd可以?jin)。当然如何调用和选择可以看看weka.gui.explorer.AttributeSelectionPanelcR?br />
上面代码的实验结果如下：(x)
result is
=== Attribute Selection on all input data ===
Search Method:
Best first.
Start set: no attributes
Search direction: forward
Stale search after 5 node expansions
Total number of subsets evaluated: 12
Merit of best subset found: 0.887
Attribute Subset Evaluator (supervised, Class (nominal): 5 class):
CFS Subset Evaluator
Including locally predictive attributes
Selected attributes: 3,4 : 2
petallength
petalwidth
old number of Attributes is 5
new number of Attributes is 2
2
3
4
原来的iris数据集中共有4个属性（包含一个分cȝ标所以一?l_(d)(j)Q经q特征选择后，只有W?和第4两个l度的特征保留，所以新特征子集有两个维度（不包含类标，有点l，不好意思，我Lq样Q?br />
最后的2Q?Q?是属性数l的下标Q表C经q特征选择保留的属性子集是W?Q?Q?个属性?br />
]]>
]]>