??xml version="1.0" encoding="utf-8" standalone="yes"?>亚洲精品伦理在线,91麻豆精品国产91久久久资源速度 ,国产在线一区不卡http://www.aygfsteel.com/robbie/category/36465.htmlzh-cnMon, 15 Dec 2008 21:48:22 GMTMon, 15 Dec 2008 21:48:22 GMT60随笔http://www.aygfsteel.com/robbie/archive/2008/12/14/246274.htmlThe MatrixThe MatrixSun, 14 Dec 2008 14:13:00 GMThttp://www.aygfsteel.com/robbie/archive/2008/12/14/246274.htmlhttp://www.aygfsteel.com/robbie/comments/246274.htmlhttp://www.aygfsteel.com/robbie/archive/2008/12/14/246274.html#Feedback0http://www.aygfsteel.com/robbie/comments/commentRss/246274.htmlhttp://www.aygfsteel.com/robbie/services/trackbacks/246274.html    q个星期的后半周主要搞了kettle的试验，做了两个例子出来Q在后箋工作中这两个例子应该也能z上用场Q本来以为kettle的文档不多，后来单独下蝲了kettle的doc压羃包，发觉里面的内容还是不的Q真要将kettle搞熟的话Q这些文还是需要仔l研M番的。另外kettle doc解压后文目录挺奇怪的Q都是数字命名的目录名，不知有啥具体含义?/p>

    下周的学习重点还是要转回到JBoss Seam中了 :-)



The Matrix 2008-12-14 22:13 发表评论
]]>Kettle的第二个实践--数据获取q{?/title><link>http://www.aygfsteel.com/robbie/archive/2008/12/14/246271.html</link><dc:creator>The Matrix</dc:creator><author>The Matrix</author><pubDate>Sun, 14 Dec 2008 13:55:00 GMT</pubDate><guid>http://www.aygfsteel.com/robbie/archive/2008/12/14/246271.html</guid><wfw:comment>http://www.aygfsteel.com/robbie/comments/246271.html</wfw:comment><comments>http://www.aygfsteel.com/robbie/archive/2008/12/14/246271.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.aygfsteel.com/robbie/comments/commentRss/246271.html</wfw:commentRss><trackback:ping>http://www.aygfsteel.com/robbie/services/trackbacks/246271.html</trackback:ping><description><![CDATA[<p><strong><font color="#0080c0" size="4">需求：</font></strong></p> <p><font color="#800080">kettletest1数据库中有table_source数据表，l构如下Q?/font></p> <ol> <li>Id                主键 <li>t_id             数据旉 <li>part_id        实例ID <li>yg               数据字段1 <li>wg              数据字段2 </li></ol> <p>该表中的数据对于不同的实例IDQ一分钟一条数据，t_id字段表示数据的时__到分钟?/p> <p><font color="#800080"> kettletest2数据库中有table_target数据表，l构如下Q?/font></p> <ol> <li>Id                     主键 <li>marketdate      数据日期Q格式ؓ yyyy-MM-dd <li>pointtime        旉Q格式ؓ HH:mm <li>pointnumber   旉的数字表C，00:01表示?Q?0:00表示?440 <li>plantcode        实例Code <li>yg                    数据字段1 <li>wg                   数据字段2 </li></ol> <p><font color="#800080">需定期table_source表中的数据获取至table_target表中Qƈq行如下处理Q?/font></p> <blockquote> <p>1、将t_id数据旉字段拆分Z个字D，分别为marketdate、pointtime、pointnumber?</p></blockquote> <blockquote> <p>        a、marketdate取t_id的日期部分?<br>        b、pointtime取t_id的时间部分?<br>        c、pointnumber为时间的数字表示Q等于hour*60+minute?<br>        d、但当t_id的时间ؓ某日?0:00Ӟ需其转化?4:00Qƈ且marketdate需取日期的前一天。如t_id?008-12-04 00:00Q则marketdate?008-12-03Qpointtime?4:00Qpointnumber?440?</p></blockquote> <blockquote> <p>2、将part_id字段映射为plantcode字段QƈҎ如下规则q行转换Q?<br>        part_id                 plantcode <br>         3206                        P01 <br>         3207                        P02 <br>         3208                        P03 </p></blockquote> <p>试中用的数据库均为mysql数据库?/p> <p><strong><font color="#0080c0" size="4">实战Q?/font></strong></p> <p>整个转换工作共分Z个步骤，如下图：</p> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/job.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; margin: 0px 0px 0px 50px; border-right-width: 0px" height="144" alt="job" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/job_thumb.jpg" width="454" border="0"></a></p> <blockquote> <p>1、定义需获取的数据的日期</p></blockquote> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans1.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; margin: 0px 0px 0px 50px; border-right-width: 0px" height="450" alt="trans1" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans1_thumb.jpg" width="600" border="0"></a> </p> <blockquote> <p>2、删除table_target表中已有数据Q?font color="#ff0000"><strong>注意一定要“执行SQl语句”面板中的“变量替换”要选上</strong></font>Q否则SQL语句中的变量不会被替换，我刚开始没注意到这个地方，N题找了半天?/p></blockquote> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans2.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; margin: 0px 0px 0px 50px; border-right-width: 0px" height="571" alt="trans2" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans2_thumb.jpg" width="500" border="0"></a> </p> <blockquote> <p>3、获取table_source中的数据Qƈ其插入table_target?/p></blockquote> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3_1.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; margin: 0px 0px 0px 50px; border-right-width: 0px" height="272" alt="trans3" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3_thumb_1.jpg" width="364" border="0"></a> </p> <blockquote> <p>        3-1、获取table_source表的数据</p></blockquote> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-1.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; margin: 0px 0px 0px 80px; border-right-width: 0px" height="356" alt="trans3-1" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-1_thumb.jpg" width="450" border="0"></a> </p> <blockquote> <p>        3-2、值映?/p></blockquote> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-2.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; margin: 0px 0px 0px 80px; border-right-width: 0px" height="299" alt="trans3-2" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-2_thumb.jpg" width="400" border="0"></a> </p> <blockquote> <p>        3-3、字D选择</p></blockquote> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-3.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; margin: 0px 0px 0px 80px; border-right-width: 0px" height="292" alt="trans3-3" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-3_thumb.jpg" width="400" border="0"></a> </p> <blockquote> <p>        3-4、对t_id字段q行处理Q增加了pointnumber字段?font color="#ff0000"><strong>在这一步骤中发现kettle的一个bugQ就是不能在JavaScript中用str2date函数Q?/strong></font>错误的具体信息参见：<a title="http://jira.pentaho.com/browse/PDI-1827" >http://jira.pentaho.com/browse/PDI-1827</a>。这个问题也折腾了好长时_刚开始怎么也想不通这个函C用时怎么会报错呢Q后来只好从字符串中截取q、月、日信息?/p> <p>       <font color="#ff0000"><strong> 该步骤中q存在另外一个人困惑的问题Q就是点几Z测试脚本”按钮，会报错，但是执行job和transformation时则不会报错?/strong></font></p></blockquote> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-4_1.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; margin: 0px 0px 0px 80px; border-right-width: 0px" height="360" alt="trans3-4" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-4_thumb_1.jpg" width="450" border="0"></a> </p> <blockquote> <p>        3-5、增加pointnumber字段臌出结果中</p></blockquote> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-5.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; margin: 0px 0px 0px 80px; border-right-width: 0px" height="292" alt="trans3-5" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-5_thumb.jpg" width="400" border="0"></a> </p> <blockquote> <p>        3-6、插入数据至table_target?/p></blockquote> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-6.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; margin: 0px 0px 0px 80px; border-right-width: 0px" height="429" alt="trans3-6" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/Kettle_1268F/trans3-6_thumb.jpg" width="400" border="0"></a> </p> <p> </p> <p>3-4步骤中的JavaScript代码如下Q?/p> <div style="border-right: #cccccc 1px solid; padding-right: 5px; border-top: #cccccc 1px solid; padding-left: 4px; font-size: 13px; padding-bottom: 4px; border-left: #cccccc 1px solid; width: 98%; padding-top: 4px; border-bottom: #cccccc 1px solid; background-color: #eeeeee"><!--<br /> <br /> Code highlighting produced by Actipro CodeHighlighter (freeware)<br /> http://www.CodeHighlighter.com/<br /> <br /> --><span style="color: #0000ff">var</span><span style="color: #000000"> pointTimeStr </span><span style="color: #000000">=</span><span style="color: #000000"> pointtime.getString();<br></span><span style="color: #0000ff">var</span><span style="color: #000000"> pointnumber </span><span style="color: #000000">=</span><span style="color: #000000"> </span><span style="color: #000000">1</span><span style="color: #000000">;<br><br></span><span style="color: #0000ff">if</span><span style="color: #000000"> (pointTimeStr </span><span style="color: #000000">==</span><span style="color: #000000"> </span><span style="color: #000000">"</span><span style="color: #000000">00:00</span><span style="color: #000000">"</span><span style="color: #000000">) {<br><br>    </span><span style="color: #0000ff">var</span><span style="color: #000000"> marketDateStr </span><span style="color: #000000">=</span><span style="color: #000000"> marketdate.getString();<br>    </span><span style="color: #0000ff">var</span><span style="color: #000000"> marketDateYear </span><span style="color: #000000">=</span><span style="color: #000000"> substr(marketDateStr, </span><span style="color: #000000">0</span><span style="color: #000000">, </span><span style="color: #000000">4</span><span style="color: #000000">);<br>    </span><span style="color: #0000ff">var</span><span style="color: #000000"> marketDateMonth </span><span style="color: #000000">=</span><span style="color: #000000"> str2num(substr(marketDateStr, </span><span style="color: #000000">5</span><span style="color: #000000">, </span><span style="color: #000000">2</span><span style="color: #000000">))</span><span style="color: #000000">-</span><span style="color: #000000">1</span><span style="color: #000000">;<br>    </span><span style="color: #0000ff">var</span><span style="color: #000000"> marketDateDay </span><span style="color: #000000">=</span><span style="color: #000000"> substr(marketDateStr, </span><span style="color: #000000">8</span><span style="color: #000000">, </span><span style="color: #000000">2</span><span style="color: #000000">);<br><br>    </span><span style="color: #0000ff">var</span><span style="color: #000000"> date </span><span style="color: #000000">=</span><span style="color: #000000"> </span><span style="color: #0000ff">new</span><span style="color: #000000"> Date();<br>    date.setYear(marketDateYear);<br>    date.setMonth(marketDateMonth);<br>    date.setDate(marketDateDay);<br></span><span style="color: #000000"><br>    </span><span style="color: #0000ff">var</span><span style="color: #000000"> temp1 </span><span style="color: #000000">=</span><span style="color: #000000"> dateAdd(date, </span><span style="color: #000000">"</span><span style="color: #000000">d</span><span style="color: #000000">"</span><span style="color: #000000">, </span><span style="color: #000000">-</span><span style="color: #000000">1</span><span style="color: #000000">);<br>    marketdate.setValue(date2str(temp1, </span><span style="color: #000000">"</span><span style="color: #000000">yyyy-MM-dd</span><span style="color: #000000">"</span><span style="color: #000000">));<br>    pointtime.setValue(</span><span style="color: #000000">"</span><span style="color: #000000">24:00</span><span style="color: #000000">"</span><span style="color: #000000">);<br><br>    pointnumber </span><span style="color: #000000">=</span><span style="color: #000000"> </span><span style="color: #000000">1440</span><span style="color: #000000">;<br>} </span><span style="color: #0000ff">else</span><span style="color: #000000"> {    <br>    </span><span style="color: #0000ff">var</span><span style="color: #000000"> hourStr </span><span style="color: #000000">=</span><span style="color: #000000"> pointTimeStr.substr(</span><span style="color: #000000">0</span><span style="color: #000000">, </span><span style="color: #000000">2</span><span style="color: #000000">);<br>    </span><span style="color: #0000ff">var</span><span style="color: #000000"> hour </span><span style="color: #000000">=</span><span style="color: #000000"> str2num(hourStr);<br>    </span><span style="color: #0000ff">var</span><span style="color: #000000"> minuteStr </span><span style="color: #000000">=</span><span style="color: #000000"> pointTimeStr.substr(</span><span style="color: #000000">3</span><span style="color: #000000">, </span><span style="color: #000000">5</span><span style="color: #000000">);<br>    </span><span style="color: #0000ff">var</span><span style="color: #000000"> minute </span><span style="color: #000000">=</span><span style="color: #000000"> str2num(minuteStr);<br>    <br>    pointnumber </span><span style="color: #000000">=</span><span style="color: #000000"> hour </span><span style="color: #000000">*</span><span style="color: #000000"> </span><span style="color: #000000">60</span><span style="color: #000000"> </span><span style="color: #000000">+</span><span style="color: #000000"> minute;<br>}</span></div> <p> </p> <p>xQ整个{换工作完成，结一下：</p> <p>    如果对kettle{etl工具比较熟悉的话Q用etl工具q行数据转换、抽取等事情q是比较方便的，比v写程序还是有优势的。但是这个{换过E中遇到的kettle的两个bug比较让h头疼Q觉得kettle好像q不是很E_?/p><img src ="http://www.aygfsteel.com/robbie/aggbug/246271.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.aygfsteel.com/robbie/" target="_blank">The Matrix</a> 2008-12-14 21:55 <a href="http://www.aygfsteel.com/robbie/archive/2008/12/14/246271.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>Kettle的第一个实?-从FTP上取文gQ再放至另一个FTP?/title><link>http://www.aygfsteel.com/robbie/archive/2008/12/12/245962.html</link><dc:creator>The Matrix</dc:creator><author>The Matrix</author><pubDate>Fri, 12 Dec 2008 07:20:00 GMT</pubDate><guid>http://www.aygfsteel.com/robbie/archive/2008/12/12/245962.html</guid><wfw:comment>http://www.aygfsteel.com/robbie/comments/245962.html</wfw:comment><comments>http://www.aygfsteel.com/robbie/archive/2008/12/12/245962.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.aygfsteel.com/robbie/comments/commentRss/245962.html</wfw:commentRss><trackback:ping>http://www.aygfsteel.com/robbie/services/trackbacks/245962.html</trackback:ping><description><![CDATA[<p>q个实践其实不难Q主要是有一个地方要注意Q就是文件名通配W的写法Q如果文件名格式为“TRANS_yyyymmdd.txt”，如TRANS_20081101.txt。如果想匚w所有以TRANS开头的文本文gQ在kettle中要写成q样QTRANS_.*[0-9].txt?/p> <p>最后在windows操作pȝ中配|定时Q务就可以定期执行该Job了?/p> <p>Job的图Q?/p> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/KettleFTPFTP_D725/total.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; border-right-width: 0px" height="208" alt="total" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/KettleFTPFTP_D725/total_thumb.jpg" width="460" border="0"></a> </p> <p>FTP配置信息Q?/p> <p><a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/KettleFTPFTP_D725/get%20ftp%20config.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; border-right-width: 0px" height="480" alt="get ftp config" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/KettleFTPFTP_D725/get%20ftp%20config_thumb.jpg" width="389" border="0"></a> <a href="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/KettleFTPFTP_D725/put%20ftp%20config.jpg"><img style="border-top-width: 0px; border-left-width: 0px; border-bottom-width: 0px; border-right-width: 0px" height="480" alt="put ftp config" src="http://www.aygfsteel.com/images/blogjava_net/robbie/WindowsLiveWriter/KettleFTPFTP_D725/put%20ftp%20config_thumb.jpg" width="453" border="0"></a></p><img src ="http://www.aygfsteel.com/robbie/aggbug/245962.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.aygfsteel.com/robbie/" target="_blank">The Matrix</a> 2008-12-12 15:20 <a href="http://www.aygfsteel.com/robbie/archive/2008/12/12/245962.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>Kettle初探http://www.aygfsteel.com/robbie/archive/2008/12/04/244464.htmlThe MatrixThe MatrixThu, 04 Dec 2008 14:48:00 GMThttp://www.aygfsteel.com/robbie/archive/2008/12/04/244464.htmlhttp://www.aygfsteel.com/robbie/comments/244464.htmlhttp://www.aygfsteel.com/robbie/archive/2008/12/04/244464.html#Feedback0http://www.aygfsteel.com/robbie/comments/commentRss/244464.htmlhttp://www.aygfsteel.com/robbie/services/trackbacks/244464.html        今天早上在网上看Ckettle发布了最新的版本Q忽然想hq其实做了不工作应该是ETL工具的拿手好戏，赶紧下蝲下来看看Q看是否能够在实际的工作中应用v来?/p>

        Z讲一下，为啥看到kettle会两眼发光?/p>

        最q写了好几个程序，用于从一个ftp去获取数据，然后转发臛_一个ftp去，或者是从一个数据库获取数据然后保存x地的数据库中Q用的是jdk中的Timer实现的定时调度，本来也没什么问题，q箋q行几个月都不会出错?/p>

        可是最q网l不是太好，周期性抽风，ping包时Q每5分钟大概会丢7-8个包Q从而导致程序也会假死，q一D|间后׃正常q活了，估计是因为用了数据库q接池的问题Q要是每ơ发h据库q接可能׃会有问题了，h也不x了，因ؓ|络最l肯定是会修好的 :-) 但是惌试ETL工具Q因为后面还有一些类似的东西要处理，不想写代码了Q用别h的轮子感觉比较好Q呵?/p>

        首先下蝲了kettle的最新版Qkettle3.1Q解压后卛_q行Q一般的开发h员稍微摸索一下，看看例子单的转换q是会做的，今天试了一把，有几个注意点C来?/p>

  1. 使用资源库（repositoryQ登录时Q默认的用户名和密码是admin/admin
  2. 当job是存攑֜资源库（一般资源库都用数据库Q中Ӟ使用Kitchen.bat执行jobӞ需使用如下的命令行Q?font color="#0080c0">
    Kitchen.bat /rep kettle /user admin /pass admin /job job?/font>
  3. 当job没有存放在资源库而存攑֜文gpȝӞ使用Kitchen.bat执行jobӞ需使用如下的命令行Q?font color="#0080c0">
    Kitchen.bat /norep /file user-transfer-job.kjb
  4. 可以使用命o行执行job后，可以用windows或linux的Q务调度来定时执行d?/li>

        在一开始用命令行方式执行jobӞL报如下的错误Q琢了好长旉ȝ整明白正的方式了?/p>

Unexpected error during transformation metadata load
No repository defined!

        下一步准备按照实际情况定制JobQ做好了再写结?/p>

The Matrix 2008-12-04 22:48 发表评论
]]> վ֩ģ壺 Զ| | | 㺺| С| ˮ| | ཭| | ͼ| ƽ̶| 踽| 书| ¹| | | | | «| | Ǭ| Ȫ| ˳| | | Դ| | ԭ| | ƽ| | ˮ| | | ȷ| ˫Ѽɽ| | ͼʲ| Ȫ| | |