k-分割交差検証とはトレーニングデータをk分割(データを”k”個に分ける)
k個から1個引いた個数を学習用に使用し、
引いた1個はテストに使用する
10個に分割すれば9個を学習用1個をテスト用にするという事
これによりk個の学習モデルとk個の性能テストができるので
k回の学習とテストを行い、それらの平均を算出する
これによってホールドアウト法よりバリアンス（過学習）を低く抑えられる
通常kの値は10前後になり、データが多い場合はkの値を増やした方が良い
但し、k値を大きくし過ぎるとバリアンス（過学習）が高くなり、
逆にk値を低くしすぎるとバイアス（学習不足）が高くなる可能性がある
機械学習上の問題に「過学習」（学習し過ぎ）というものがあり
この状態をバリアンスが高いと表現し、
その反対に「学習不足」というものもあり
こちらはバイアスが高いと表現する
過学習を防ぐための手段にホールドアウト法というものがある
この方法によって学習データをトレーニングデータとテストデータに分割
トレーニングデータは文言から想像できるように学習という意味で
テストデータというのは学習した内容の評価となる
トレーニングデータという概念は更に分割され、
トレーニングデータサブセットと
検証データセットという２つになります。
これはテストデータがトレーニングデータの一部になってしまう事を防ぐ為にある
トレーニングデータサブセットと検証データセットの２つで学習を行い
もう一方のテストデータを用いて最終テストを行う
このホールドアウト法の他に
「k分割交差検証」と呼ばれるものもある
DataFrame型の変数に対して、変数.diff(変数.diff - x, axis="0 or 1")と指定で
行間または列間の差を計算したDataFrameが作成
第1引数が正の場合は前の行との差、負の場合は後の行との差
axisは0の場合が行(横)方向、1の場合が列(縦)方向です。
|
1 |
<span role="presentation"><span class="cm-keyword">import</span> <span class="cm-variable">numpy</span> <span class="cm-keyword">as</span> <span class="cm-variable">np</span></span> |
|
1 |
<span role="presentation"><span class="cm-keyword">import</span> <span class="cm-variable">pandas</span> <span class="cm-keyword">as</span> <span class="cm-variable">pd</span></span> |
|
1 |
<span role="presentation"><span class="cm-variable">np</span>.<span class="cm-property">random</span>.<span class="cm-property">seed</span>(<span class="cm-number">0</span>)</span> |
|
1 |
<span role="presentation"><span class="cm-variable">columns</span> = [<span class="cm-string">"apple"</span>, <span class="cm-string">"orange"</span>, <span class="cm-string">"banana"</span>, <span class="cm-string">"strawberry"</span>, <span class="cm-string">"kiwifruit"</span>]</span> |
|
1 |
<span role="presentation"></span> |
|
1 |
<span role="presentation"><span class="cm-comment"># DataFrameを生成し、列を追加</span></span> |
|
1 |
<span role="presentation"><span class="cm-variable">df</span> = <span class="cm-variable">pd</span>.<span class="cm-property">DataFrame</span>()</span> |
|
1 |
<span role="presentation"><span class="cm-keyword">for</span> <span class="cm-variable">column</span> <span class="cm-keyword">in</span> <span class="cm-variable">columns</span>:</span> |
|
1 |
<span role="presentation"> <span class="cm-variable">df</span>[<span class="cm-variable">column</span>] = <span class="cm-variable">np</span>.<span class="cm-property">random</span>.<span class="cm-property">choice</span>(<span class="cm-builtin">range</span>(<span class="cm-number">1</span>, <span class="cm-number">11</span>), <span class="cm-number">10</span>)</span> |
|
1 |
<span role="presentation"><span class="cm-variable">df</span>.<span class="cm-property">index</span> = <span class="cm-builtin">range</span>(<span class="cm-number">1</span>, <span class="cm-number">11</span>)</span> |
|
1 |
<span role="presentation"></span> |
|
1 |
<span role="presentation"><span class="cm-comment"># dfの各行について、2行後の行との差を計算したDataFrameをdf_diffに代入</span></span> |
|
1 2 3 |
<span role="presentation"><span class="cm-variable">df_diff</span>=<span class="cm-variable">df</span>.<span class="cm-property">diff</span> <span class=" CodeMirror-matchingbracket">(</span><span class="cm-operator">-</span><span class="cm-number">2</span>,<span class="cm-variable">axis</span> = <span class="cm-number">0</span><span class=" CodeMirror-matchingbracket">) #第一引数がマイナスの場合は下、右方向へ処理 第一引数がプラスの場合は上、左方向へ処理</span></span> |
|
1 |
</code><code> |
|
1 |
<span role="presentation"><span class="cm-builtin">print</span>(<span class="cm-variable">df</span>)</span> |
|
1 |
<span role="presentation"><span class="cm-builtin">print</span>(<span class="cm-variable">df_diff</span>)</span> |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
apple orange banana strawberry kiwifruit 1 6 8 6 3 10 2 1 7 10 4 10 3 4 9 9 9 1 4 4 9 10 2 5 5 8 2 5 4 8 6 10 7 4 4 4 7 4 8 1 4 3 8 6 8 4 8 8 9 3 9 6 1 3 10 5 2 1 2 1 apple orange banana strawberry kiwifruit 1 2.0 -1.0 -3.0 -6.0 9.0 2 -3.0 -2.0 0.0 2.0 5.0 3 -4.0 7.0 4.0 5.0 -7.0 4 -6.0 2.0 6.0 -2.0 1.0 5 4.0 -6.0 4.0 0.0 5.0 6 4.0 -1.0 0.0 -4.0 -4.0 7 1.0 -1.0 -5.0 3.0 0.0 8 1.0 6.0 3.0 6.0 7.0 9 NaN NaN NaN NaN NaN 10 NaN NaN NaN NaN NaN |
列ごとの平均値、最大値、最小値等統計的情報を要約統計量と呼ぶ。
DataFrameの変数に対して、変数.describe()は変数の列ごとの
個数、平均値、標準偏差、最小値、四分位数、最大値を返す。
DataFrameのインデックスの数字は統計量の名前に置き換わる
|
1 |
<span role="presentation"><span class="cm-keyword">import</span> <span class="cm-variable">numpy</span> <span class="cm-keyword">as</span> <span class="cm-variable">np</span></span> |
|
1 |
<span role="presentation"><span class="cm-keyword">import</span> <span class="cm-variable">pandas</span> <span class="cm-keyword">as</span> <span class="cm-variable">pd</span></span> |
|
1 |
<span role="presentation"><span class="cm-variable">np</span>.<span class="cm-property">random</span>.<span class="cm-property">seed</span>(<span class="cm-number">0</span>)</span> |
|
1 |
<span role="presentation"><span class="cm-variable">columns</span> = [<span class="cm-string">"apple"</span>, <span class="cm-string">"orange"</span>, <span class="cm-string">"banana"</span>, <span class="cm-string">"strawberry"</span>, <span class="cm-string">"kiwifruit"</span>]</span> |
|
1 |
<span role="presentation"></span> |
|
1 |
<span role="presentation"><span class="cm-comment"># DataFrameを生成し、列を追加</span></span> |
|
1 |
<span role="presentation"><span class="cm-variable">df</span> = <span class="cm-variable">pd</span>.<span class="cm-property">DataFrame</span>()</span> |
|
1 |
<span role="presentation"><span class="cm-keyword">for</span> <span class="cm-variable">column</span> <span class="cm-keyword">in</span> <span class="cm-variable">columns</span>:</span> |
|
1 |
<span role="presentation"> <span class="cm-variable">df</span>[<span class="cm-variable">column</span>] = <span class="cm-variable">np</span>.<span class="cm-property">random</span>.<span class="cm-property">choice</span>(<span class="cm-builtin">range</span>(<span class="cm-number">1</span>, <span class="cm-number">11</span>), <span class="cm-number">10</span>)</span> |
|
1 |
<span role="presentation"><span class="cm-variable">df</span>.<span class="cm-property">index</span> = <span class="cm-builtin">range</span>(<span class="cm-number">1</span>, <span class="cm-number">11</span>)</span> |
|
1 |
<span role="presentation"><span class="cm-builtin">print</span><span class=" CodeMirror-matchingbracket">(</span><span class="cm-variable">df</span><span class=" CodeMirror-matchingbracket">)</span></span> |
|
1 |
<span role="presentation"><span class="cm-comment"># dfの要約統計量のうち、"mean", "max", "min"を取り出してdf_desに代入してください</span></span> |
|
1 |
<span role="presentation"></span> |
|
1 |
<span role="presentation"><span class="cm-variable">df_des</span> = <span class="cm-variable">df</span>.<span class="cm-property">describe</span>().<span class="cm-property">loc</span>[[<span class="cm-string">"mean"</span>, <span class="cm-string">"max"</span>, <span class="cm-string">"min"</span>]]</span> |
|
1 |
<span role="presentation"><span class="cm-builtin">print</span>(<span class="cm-variable">df_des</span>)</span> |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
apple orange banana strawberry kiwifruit 1 6 8 6 3 10 2 1 7 10 4 10 3 4 9 9 9 1 4 4 9 10 2 5 5 8 2 5 4 8 6 10 7 4 4 4 7 4 8 1 4 3 8 6 8 4 8 8 9 3 9 6 1 3 10 5 2 1 2 1 ↓ apple orange banana strawberry kiwifruit mean 5.1 6.9 5.6 4.1 5.3 max 10.0 9.0 10.0 9.0 10.0 min 1.0 2.0 1.0 1.0 1.0 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
import pandas as pd # 注文情報 order_df = pd.DataFrame([[1000, 2546, 103], [1001, 4352, 101], [1002, 342, 101]], columns=["id", "item_id", "customer_id"]) # 顧客情報 customer_df = pd.DataFrame([[101, "Tanaka"], [102, "Suzuki"], [103, "Kato"]], columns=["id", "name"]) # customer_dfを元に"name"をorder_dfに結合してorder_dfに代入してください order_df = pd.merge(order_df,customer_df,left_on="customer_id",right_on="id",how="inner") print(order_df) |
|
1 2 3 4 |
id_x item_id customer_id id_y name 0 1000 2546 103 103 Kato 1 1001 4352 101 101 Tanaka 2 1002 342 101 101 Tanaka |
DataFrameの結合で共通するものが無くても結合し、
数値の孫座しない部分はNoneとなる
|
1 2 3 |
<span role="presentation"><span class="cm-keyword"> import</span> <span class="cm-variable">numpy</span> <span class="cm-keyword">as</span> <span class="cm-variable">np</span></span> |
|
1 |
<span role="presentation"><span class="cm-keyword">import</span> <span class="cm-variable">pandas</span> <span class="cm-keyword">as</span> <span class="cm-variable">pd</span></span> |
|
1 |
<span role="presentation"></span> |
|
1 |
<span role="presentation"><span class="cm-variable">data1</span> = {<span class="cm-string">"fruits"</span>: [<span class="cm-string">"apple"</span>, <span class="cm-string">"orange"</span>, <span class="cm-string">"banana"</span>, <span class="cm-string">"strawberry"</span>, <span class="cm-string">"kiwifruit"</span>],</span> |
|
1 |
<span role="presentation"> <span class="cm-string">"year"</span>: [<span class="cm-number">2001</span>, <span class="cm-number">2002</span>, <span class="cm-number">2001</span>, <span class="cm-number">2008</span>, <span class="cm-number">2006</span>],</span> |
|
1 |
<span role="presentation"> <span class="cm-string">"amount"</span>: [<span class="cm-number">1</span>, <span class="cm-number">4</span>, <span class="cm-number">5</span>, <span class="cm-number">6</span>, <span class="cm-number">3</span>]}</span> |
|
1 |
<span role="presentation"><span class="cm-variable">df1</span> = <span class="cm-variable">pd</span>.<span class="cm-property">DataFrame</span>(<span class="cm-variable">data1</span>)</span> |
|
1 |
<span role="presentation"></span> |
|
1 |
<span role="presentation"><span class="cm-variable">data2</span> = {<span class="cm-string">"fruits"</span>: [<span class="cm-string">"apple"</span>, <span class="cm-string">"orange"</span>, <span class="cm-string">"banana"</span>, <span class="cm-string">"strawberry"</span>, <span class="cm-string">"mango"</span>],</span> |
|
1 |
<span role="presentation"> <span class="cm-string">"year"</span>: [<span class="cm-number">2001</span>, <span class="cm-number">2002</span>, <span class="cm-number">2001</span>, <span class="cm-number">2008</span>, <span class="cm-number">2007</span>],</span> |
|
1 |
<span role="presentation"> <span class="cm-string">"price"</span>: [<span class="cm-number">150</span>, <span class="cm-number">120</span>, <span class="cm-number">100</span>, <span class="cm-number">250</span>, <span class="cm-number">3000</span>]}</span> |
|
1 |
<span role="presentation"><span class="cm-variable">df2</span> = <span class="cm-variable">pd</span>.<span class="cm-property">DataFrame</span>(<span class="cm-variable">data2</span>)</span> |
|
1 |
<span role="presentation"></span> |
|
1 |
<span role="presentation"><span class="cm-comment"># df1, df2の中身を確認してください</span></span> |
|
1 |
<span role="presentation"><span class="cm-builtin">print</span>(<span class="cm-variable">df1</span>)</span> |
|
1 |
<span role="presentation"><span class="cm-builtin">print</span>(<span class="cm-variable">df2</span>)</span> |
|
1 |
<span role="presentation"></span> |
|
1 |
<span role="presentation"><span class="cm-comment"># df1とdf2を"fruits"をキーに外部結合して作成したDataFrameをdf3に代入してください</span></span> |
|
1 |
<span role="presentation"><span class="cm-variable">df3</span> = <span class="cm-variable">pd</span>.<span class="cm-property">merge</span>(<span class="cm-variable">df1</span>, <span class="cm-variable">df2</span>, <span class="cm-variable">on</span>=<span class="cm-string">"fruits"</span>, <span class="cm-variable">how</span>=<span class="cm-string">"outer"</span>)</span> |
|
1 |
<span role="presentation"></span> |
|
1 |
<span role="presentation"><span class="cm-builtin">print</span><span class=" CodeMirror-matchingbracket">(</span><span class="cm-variable">df3</span><span class=" CodeMirror-matchingbracket">)</span></span> |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
amount fruits year 0 1 apple 2001 1 4 orange 2002 2 5 banana 2001 3 6 strawberry 2008 4 3 kiwifruit 2006 fruits price year 0 apple 150 2001 1 orange 120 2002 2 banana 100 2001 3 strawberry 250 2008 4 mango 3000 2007 amount fruits year_x price year_y 0 1.0 apple 2001.0 150.0 2001.0 1 4.0 orange 2002.0 120.0 2002.0 2 5.0 banana 2001.0 100.0 2001.0 3 6.0 strawberry 2008.0 250.0 2008.0 4 3.0 kiwifruit 2006.0 NaN NaN 5 NaN mango NaN 3000.0 2007.0 |