{ "cells": [ { "cell_type": "markdown", "id": "51fc55d1", "metadata": {}, "source": [ "# Capstone 1 — Session 1: Data Import and Cleaning\n", "\n", "**Run timestamp:** `2026-02-19 01:49:22`\n", "\n", "## Goal\n", "- I inspected and validated the NSMES1988 dataset, documented structure/quality findings, and prepared the cleaned handoff file for the next capstone.\n", "- I also generated required export artifacts and captured evidence for rows, columns, dtypes, missingness, value ranges, and memory usage.\n", "\n", "## Inputs\n", "- `NSMES1988.csv`\n", "\n", "## Outputs\n", "- All exports go to `./outputs/` (and plots to `./outputs/plots/` when applicable)\n", "\n", "## Libraries (documented)\n", "- `pandas`: I used it for loading, profiling, validation checks, and CSV/JSON exports.\n", "\n", "## Key dataset note\n", "- `age` is encoded as **Age in years (divided by 10)** (e.g., `6.9` = 69 years)." ] }, { "cell_type": "markdown", "id": "ba6002f0", "metadata": {}, "source": [ "## C?-T0 — Runtime setup (paths + output folders)\n", "I initialized paths and output folders in this setup block to keep the workflow reproducible." ] }, { "cell_type": "code", "execution_count": null, "id": "74bfbf52", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Capstone: 1 | Session: Session 1: Data Import and Cleaning\n", "Run timestamp: 2026-02-19 00:30:45\n", "Base directory: c:\\DEV_Projects\\SIMPLILEARN\\CAPSTONE_Applied_Data_Science_with _Python\\Incremental_Capstone\\Capstone 1\n", "Dataset path: c:\\DEV_Projects\\SIMPLILEARN\\CAPSTONE_Applied_Data_Science_with _Python\\Incremental_Capstone\\Capstone 1\\NSMES1988.csv\n", "Output directory: c:\\DEV_Projects\\SIMPLILEARN\\CAPSTONE_Applied_Data_Science_with _Python\\Incremental_Capstone\\Capstone 1\\outputs\n", "Plots directory: c:\\DEV_Projects\\SIMPLILEARN\\CAPSTONE_Applied_Data_Science_with _Python\\Incremental_Capstone\\Capstone 1\\outputs\\plots\n" ] } ], "source": [ "from pathlib import Path\n", "from datetime import datetime\n", "from urllib.parse import quote\n", "\n", "try:\n", " from IPython.display import display\n", "except Exception:\n", " def display(value):\n", " print(value)\n", "\n", "CAPSTONE = 1\n", "SESSION_TITLE = 'Session 1: Data Import and Cleaning'\n", "IS_COLAB = 'google.colab' in __import__('sys').modules\n", "GITHUB_REPO_OWNER = 'FrancisBurnet'\n", "GITHUB_REPO_NAME = 'francisburnet'\n", "GITHUB_REPO_BRANCH = 'main'\n", "CAPSTONE_ROOT = Path('Incremental Capstones/Applied Data Science with Python/Capstone 1')\n", "DATASET_FILENAME = 'NSMES1988.csv'\n", "\n", "\n", "def build_raw_github_url(relative_path: Path) -> str:\n", " encoded_path = quote(relative_path.as_posix(), safe='/')\n", " return (\n", " f\"https://raw.githubusercontent.com/{GITHUB_REPO_OWNER}/{GITHUB_REPO_NAME}/\"\n", " f\"{GITHUB_REPO_BRANCH}/{encoded_path}\"\n", " )\n", "\n", "\n", "def resolve_capstone_dir() -> Path | None:\n", " current = Path.cwd().resolve()\n", " capstone_parts = CAPSTONE_ROOT.parts\n", " for candidate in [current, *current.parents]:\n", " if len(candidate.parts) >= len(capstone_parts) and candidate.parts[-len(capstone_parts):] == capstone_parts:\n", " return candidate\n", " nested_candidate = candidate / CAPSTONE_ROOT\n", " if nested_candidate.exists():\n", " return nested_candidate\n", " return None\n", "\n", "\n", "CAPSTONE_DIR = resolve_capstone_dir()\n", "DATASET_URL = build_raw_github_url(CAPSTONE_ROOT / DATASET_FILENAME)\n", "\n", "if CAPSTONE_DIR is not None:\n", " BASE_DIR = CAPSTONE_DIR\n", " OUTPUT_MODE = 'permanent capstone outputs'\n", " OUTPUT_DISPLAY = (CAPSTONE_ROOT / 'outputs').as_posix()\n", "else:\n", " BASE_DIR = Path('/content/capstone-1-runtime') if IS_COLAB else Path.cwd().resolve() / 'capstone-1-runtime'\n", " OUTPUT_MODE = 'runtime scratch outputs'\n", " OUTPUT_DISPLAY = BASE_DIR.name + '/outputs'\n", "\n", "OUTPUT_DIR = BASE_DIR / 'outputs'\n", "PLOTS_DIR = OUTPUT_DIR / 'plots'\n", "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n", "PLOTS_DIR.mkdir(parents=True, exist_ok=True)\n", "\n", "print(f'Capstone: {CAPSTONE} | {SESSION_TITLE}')\n", "print('Run timestamp:', datetime.now().strftime('%Y-%m-%d %H:%M:%S'))\n", "print('Dataset source:', DATASET_URL)\n", "print('Output mode:', OUTPUT_MODE)\n", "print('Output target:', OUTPUT_DISPLAY)" ] }, { "cell_type": "markdown", "id": "79f6190f", "metadata": {}, "source": [ "## C?-T1 — Imports I used\n", "I documented each import with why I used it and what it enabled in the analysis." ] }, { "cell_type": "code", "execution_count": 2, "id": "65cf67e1", "metadata": {}, "outputs": [], "source": [ "import pandas as pd # DataFrames + CSV/JSON IO + analysis tables" ] }, { "cell_type": "markdown", "id": "1ed2c0f6", "metadata": {}, "source": [ "## C?-T2 — Load dataset\n", "I loaded the capstone dataset using the required default filename." ] }, { "cell_type": "code", "execution_count": null, "id": "2fadbd86", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loaded: c:\\DEV_Projects\\SIMPLILEARN\\CAPSTONE_Applied_Data_Science_with _Python\\Incremental_Capstone\\Capstone 1\\NSMES1988.csv\n", "Shape: (4406, 19)\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0visitsnvisitsovisitsnovisitsemergencyhospitalhealthchronicadlregionagegendermarriedschoolincomeemployedinsurancemedicaid
01500001average2normalother6.9maleyes62.8810yesyesno
12102020average2normalother7.4femaleyes102.7478noyesno
231300033poor4limitedother6.6femaleno100.6532nonoyes
341605011poor2limitedother7.6maleyes30.6588noyesno
45300000average2limitedother7.9femaleyes60.6588noyesno
\n", "
" ], "text/plain": [ " Unnamed: 0 visits nvisits ovisits novisits emergency hospital \\\n", "0 1 5 0 0 0 0 1 \n", "1 2 1 0 2 0 2 0 \n", "2 3 13 0 0 0 3 3 \n", "3 4 16 0 5 0 1 1 \n", "4 5 3 0 0 0 0 0 \n", "\n", " health chronic adl region age gender married school income \\\n", "0 average 2 normal other 6.9 male yes 6 2.8810 \n", "1 average 2 normal other 7.4 female yes 10 2.7478 \n", "2 poor 4 limited other 6.6 female no 10 0.6532 \n", "3 poor 2 limited other 7.6 male yes 3 0.6588 \n", "4 average 2 limited other 7.9 female yes 6 0.6588 \n", "\n", " employed insurance medicaid \n", "0 yes yes no \n", "1 no yes no \n", "2 no no yes \n", "3 no yes no \n", "4 no yes no " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "DEFAULT_DATASET = 'NSMES1988.csv'\n", "\n", "if 'pd' not in globals():\n", " import pandas as pd\n", "\n", "if 'display' not in globals():\n", " try:\n", " from IPython.display import display\n", " except Exception:\n", " def display(value):\n", " print(value)\n", "\n", "if 'DATASET_URL' not in globals():\n", " from pathlib import Path\n", " from urllib.parse import quote\n", " DATASET_URL = (\n", " f\"https://raw.githubusercontent.com/FrancisBurnet/francisburnet/main/\"\n", " f\"Incremental%20Capstones/Applied%20Data%20Science%20with%20Python/Capstone%201/{DEFAULT_DATASET}\"\n", " )\n", "\n", "df = pd.read_csv(DATASET_URL)\n", "\n", "print('Dataset source used:', DATASET_URL)\n", "print('Shape:', df.shape)\n", "display(df.head())" ] }, { "cell_type": "markdown", "id": "4ca8499b", "metadata": {}, "source": [ "## C?-T3 — Validation checks\n", "- I confirmed expected columns exist\n", "- I confirmed key dtypes\n", "- I checked missing values\n", "\n", "**Results Capture:**\n", "- What I did: I validated expected columns, reviewed dtypes, and computed missing-value counts.\n", "- What I found: the raw dataset has shape `(4406, 19)`, all required columns are present, and missing values are zero across all fields.\n", "- Caveats: `Unnamed: 0` behaves like an index-like column and was removed in the cleaned export." ] }, { "cell_type": "code", "execution_count": 4, "id": "7254f672", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Missing expected columns: []\n", "\n", "Dtypes:\n" ] }, { "data": { "text/plain": [ "Unnamed: 0 int64\n", "visits int64\n", "nvisits int64\n", "ovisits int64\n", "novisits int64\n", "emergency int64\n", "hospital int64\n", "health str\n", "chronic int64\n", "adl str\n", "region str\n", "age float64\n", "gender str\n", "married str\n", "school int64\n", "income float64\n", "employed str\n", "insurance str\n", "medicaid str\n", "dtype: object" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Missing values (count):\n" ] }, { "data": { "text/plain": [ "Unnamed: 0 0\n", "visits 0\n", "nvisits 0\n", "ovisits 0\n", "novisits 0\n", "dtype: int64" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "expected_cols = [\n", " \"Unnamed: 0\", \"visits\", \"nvisits\", \"ovisits\", \"novisits\", \"emergency\", \"hospital\",\n", " \"health\", \"chronic\", \"adl\", \"region\", \"age\", \"gender\", \"married\",\n", " \"school\", \"income\", \"employed\", \"insurance\", \"medicaid\"\n", "]\n", "\n", "missing_cols = [c for c in expected_cols if c not in df.columns]\n", "print(\"Missing expected columns:\", missing_cols)\n", "\n", "print(\"\\nDtypes:\")\n", "display(df.dtypes)\n", "\n", "print(\"\\nMissing values (count):\")\n", "na_counts = df.isna().sum().sort_values(ascending=False)\n", "display(na_counts[na_counts > 0] if (na_counts > 0).any() else na_counts.head())\n" ] }, { "cell_type": "markdown", "id": "84c91703", "metadata": {}, "source": [ "## C1-T4 — Inspection (rows/cols/dtypes) + basic profiling\n", "**PDF requirement:** Inspect the data and report details such as rows, columns, and data types.\n", "\n", "### What I completed\n", "- I profiled the dataset using `shape`, `columns`, `dtypes`, `info`, and numeric summaries.\n", "\n", "### Results Capture\n", "- `df.shape = (4406, 19)`.\n", "- I identified 11 numeric and 8 non-numeric columns.\n", "- I flagged `Unnamed: 0` as index-like and removed it in the final cleaned export.\n", "\n", "### Code evidence\n", "- The next cell shows the exact profiling code I ran." ] }, { "cell_type": "code", "execution_count": 5, "id": "084f60ba", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Shape: (4406, 19)\n", "\n", "Columns:\n", "['Unnamed: 0', 'visits', 'nvisits', 'ovisits', 'novisits', 'emergency', 'hospital', 'health', 'chronic', 'adl', 'region', 'age', 'gender', 'married', 'school', 'income', 'employed', 'insurance', 'medicaid']\n", "\n", "Info:\n", "\n", "RangeIndex: 4406 entries, 0 to 4405\n", "Data columns (total 19 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 Unnamed: 0 4406 non-null int64 \n", " 1 visits 4406 non-null int64 \n", " 2 nvisits 4406 non-null int64 \n", " 3 ovisits 4406 non-null int64 \n", " 4 novisits 4406 non-null int64 \n", " 5 emergency 4406 non-null int64 \n", " 6 hospital 4406 non-null int64 \n", " 7 health 4406 non-null str \n", " 8 chronic 4406 non-null int64 \n", " 9 adl 4406 non-null str \n", " 10 region 4406 non-null str \n", " 11 age 4406 non-null float64\n", " 12 gender 4406 non-null str \n", " 13 married 4406 non-null str \n", " 14 school 4406 non-null int64 \n", " 15 income 4406 non-null float64\n", " 16 employed 4406 non-null str \n", " 17 insurance 4406 non-null str \n", " 18 medicaid 4406 non-null str \n", "dtypes: float64(2), int64(9), str(8)\n", "memory usage: 654.1 KB\n", "\n", "Describe (numeric):\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0visitsnvisitsovisitsnovisitsemergencyhospitalchronicageschoolincome
count4406.0000004406.0000004406.0000004406.0000004406.0000004406.0000004406.0000004406.0000004406.0000004406.0000004406.000000
mean2203.5000005.7743991.6180210.7507940.5360870.2635040.2959601.5419887.40240610.2902862.527132
std1272.0469726.7592255.3170563.6527593.8795060.7036590.7463981.3496320.6334053.7387362.924648
min1.0000000.0000000.0000000.0000000.0000000.0000000.0000000.0000006.6000000.000000-1.012500
25%1102.2500001.0000000.0000000.0000000.0000000.0000000.0000001.0000006.9000008.0000000.912150
50%2203.5000004.0000000.0000000.0000000.0000000.0000000.0000001.0000007.30000011.0000001.698150
75%3304.7500008.0000001.0000000.0000000.0000000.0000000.0000002.0000007.80000012.0000003.172850
max4406.00000089.000000104.000000141.000000155.00000012.0000008.0000008.00000010.90000018.00000054.835100
\n", "
" ], "text/plain": [ " Unnamed: 0 visits nvisits ovisits novisits \\\n", "count 4406.000000 4406.000000 4406.000000 4406.000000 4406.000000 \n", "mean 2203.500000 5.774399 1.618021 0.750794 0.536087 \n", "std 1272.046972 6.759225 5.317056 3.652759 3.879506 \n", "min 1.000000 0.000000 0.000000 0.000000 0.000000 \n", "25% 1102.250000 1.000000 0.000000 0.000000 0.000000 \n", "50% 2203.500000 4.000000 0.000000 0.000000 0.000000 \n", "75% 3304.750000 8.000000 1.000000 0.000000 0.000000 \n", "max 4406.000000 89.000000 104.000000 141.000000 155.000000 \n", "\n", " emergency hospital chronic age school \\\n", "count 4406.000000 4406.000000 4406.000000 4406.000000 4406.000000 \n", "mean 0.263504 0.295960 1.541988 7.402406 10.290286 \n", "std 0.703659 0.746398 1.349632 0.633405 3.738736 \n", "min 0.000000 0.000000 0.000000 6.600000 0.000000 \n", "25% 0.000000 0.000000 1.000000 6.900000 8.000000 \n", "50% 0.000000 0.000000 1.000000 7.300000 11.000000 \n", "75% 0.000000 0.000000 2.000000 7.800000 12.000000 \n", "max 12.000000 8.000000 8.000000 10.900000 18.000000 \n", "\n", " income \n", "count 4406.000000 \n", "mean 2.527132 \n", "std 2.924648 \n", "min -1.012500 \n", "25% 0.912150 \n", "50% 1.698150 \n", "75% 3.172850 \n", "max 54.835100 " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Head:\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0visitsnvisitsovisitsnovisitsemergencyhospitalhealthchronicadlregionagegendermarriedschoolincomeemployedinsurancemedicaid
01500001average2normalother6.9maleyes62.8810yesyesno
12102020average2normalother7.4femaleyes102.7478noyesno
231300033poor4limitedother6.6femaleno100.6532nonoyes
341605011poor2limitedother7.6maleyes30.6588noyesno
45300000average2limitedother7.9femaleyes60.6588noyesno
\n", "
" ], "text/plain": [ " Unnamed: 0 visits nvisits ovisits novisits emergency hospital \\\n", "0 1 5 0 0 0 0 1 \n", "1 2 1 0 2 0 2 0 \n", "2 3 13 0 0 0 3 3 \n", "3 4 16 0 5 0 1 1 \n", "4 5 3 0 0 0 0 0 \n", "\n", " health chronic adl region age gender married school income \\\n", "0 average 2 normal other 6.9 male yes 6 2.8810 \n", "1 average 2 normal other 7.4 female yes 10 2.7478 \n", "2 poor 4 limited other 6.6 female no 10 0.6532 \n", "3 poor 2 limited other 7.6 male yes 3 0.6588 \n", "4 average 2 limited other 7.9 female yes 6 0.6588 \n", "\n", " employed insurance medicaid \n", "0 yes yes no \n", "1 no yes no \n", "2 no no yes \n", "3 no yes no \n", "4 no yes no " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Inspection\n", "print(\"Shape:\", df.shape)\n", "print(\"\\nColumns:\")\n", "print(df.columns.tolist())\n", "\n", "print(\"\\nInfo:\")\n", "df.info()\n", "\n", "print(\"\\nDescribe (numeric):\")\n", "display(df.describe())\n", "\n", "print(\"\\nHead:\")\n", "display(df.head())\n" ] }, { "cell_type": "markdown", "id": "d193b513", "metadata": {}, "source": [ "## C1-T5 — Missing values + cleanliness check\n", "**PDF requirement:** Determine if the data is clean / identify missing values.\n", "\n", "### What I completed\n", "- I calculated missing counts and percentages for every column.\n", "\n", "### Results Capture\n", "- Missing-count output shows zero null values across all columns.\n", "- No missing-value treatment was required.\n", "\n", "### Code evidence\n", "- The next cell contains the exact missingness computation and table output." ] }, { "cell_type": "code", "execution_count": 6, "id": "85207bdb", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
missing_countmissing_pct
Unnamed: 000.0
visits00.0
nvisits00.0
ovisits00.0
novisits00.0
\n", "
" ], "text/plain": [ " missing_count missing_pct\n", "Unnamed: 0 0 0.0\n", "visits 0 0.0\n", "nvisits 0 0.0\n", "ovisits 0 0.0\n", "novisits 0 0.0" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Missing values\n", "na_counts = df.isna().sum().sort_values(ascending=False)\n", "na_pct = (df.isna().mean() * 100).round(2)\n", "missing_summary = pd.DataFrame({\"missing_count\": na_counts, \"missing_pct\": na_pct})\n", "display(missing_summary[missing_summary[\"missing_count\"] > 0] if (missing_summary[\"missing_count\"] > 0).any() else missing_summary.head())\n" ] }, { "cell_type": "markdown", "id": "1d5f197d", "metadata": {}, "source": [ "## C1-T6 — Comment on age + income (datatype + range + meaning)\n", "**PDF requirement:** Comment on age and income: data type and values; determine range.\n", "\n", "### What I completed\n", "- I reviewed descriptive statistics for `age` and `income` and documented the encoded age interpretation.\n", "\n", "### Results Capture\n", "- `age` dtype=`float64`, range=`6.6` to `10.9` (years divided by 10).\n", "- `income` dtype=`float64`, range=`-1.0125` to `54.8351`.\n", "- Example interpretation: `age=6.9` corresponds to 69 years.\n", "\n", "### Code evidence\n", "- The next cell contains the column-level descriptive output used for this interpretation." ] }, { "cell_type": "code", "execution_count": 7, "id": "e5f86ebf", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "age dtype: float64\n", "count 4406.000000\n", "mean 7.402406\n", "std 0.633405\n", "min 6.600000\n", "25% 6.900000\n", "50% 7.300000\n", "75% 7.800000\n", "max 10.900000\n", "Name: age, dtype: float64\n", "\n", "income dtype: float64\n", "count 4406.000000\n", "mean 2.527132\n", "std 2.924648\n", "min -1.012500\n", "25% 0.912150\n", "50% 1.698150\n", "75% 3.172850\n", "max 54.835100\n", "Name: income, dtype: float64\n" ] } ], "source": [ "# Age + income notes\n", "for col in [\"age\", \"income\"]:\n", " if col in df.columns:\n", " print(f\"\\n{col} dtype:\", df[col].dtype)\n", " print(df[col].describe())\n", " else:\n", " print(f\"Column not found: {col}\")\n" ] }, { "cell_type": "markdown", "id": "90fed473", "metadata": {}, "source": [ "## C1-T7 — Export JSON and comment on format\n", "**PDF requirement:** Export the dataframe to JSON (NSMES1988.json) and comment on the JSON.\n", "\n", "### What I completed\n", "- I exported the dataset to JSON with `records` orientation and previewed a snippet for format confirmation.\n", "\n", "### Results Capture\n", "- Export path: `outputs/NSMES1988.json`; orientation: `records`.\n", "- The file is row-oriented and readable for downstream systems that expect object-per-record JSON.\n", "\n", "### Artifacts\n", "- `outputs/NSMES1988.json`\n", "\n", "### Code evidence\n", "- The next cell shows the export and snippet-preview code I executed." ] }, { "cell_type": "code", "execution_count": 8, "id": "03eb9771", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved: c:\\DEV_Projects\\SIMPLILEARN\\CAPSTONE_Applied_Data_Science_with _Python\\Incremental_Capstone\\Capstone 1\\outputs\\NSMES1988.json\n", "\n", "JSON snippet (first 500 chars):\n", " [{\"Unnamed: 0\":1,\"visits\":5,\"nvisits\":0,\"ovisits\":0,\"novisits\":0,\"emergency\":0,\"hospital\":1,\"health\":\"average\",\"chronic\":2,\"adl\":\"normal\",\"region\":\"other\",\"age\":6.9,\"gender\":\"male\",\"married\":\"yes\",\"school\":6,\"income\":2.881,\"employed\":\"yes\",\"insurance\":\"yes\",\"medicaid\":\"no\"},{\"Unnamed: 0\":2,\"visits\":1,\"nvisits\":0,\"ovisits\":2,\"novisits\":0,\"emergency\":2,\"hospital\":0,\"health\":\"average\",\"chronic\":2,\"adl\":\"normal\",\"region\":\"other\",\"age\":7.4,\"gender\":\"female\",\"married\":\"yes\",\"school\":10,\"income\":2.7478\n" ] } ], "source": [ "# Export JSON\n", "json_path = BASE_DIR / \"outputs\" / \"NSMES1988.json\"\n", "json_path.parent.mkdir(parents=True, exist_ok=True)\n", "\n", "df.to_json(json_path, orient=\"records\")\n", "print(\"Saved:\", json_path)\n", "\n", "# Preview first ~500 chars\n", "with open(json_path, \"r\", encoding=\"utf-8\") as f:\n", " snippet = f.read(500)\n", "print(\"\\nJSON snippet (first 500 chars):\\n\", snippet)" ] }, { "cell_type": "markdown", "id": "2e55b2fc", "metadata": {}, "source": [ "## C1-T8 — Memory usage + dtype recommendations\n", "**PDF requirement:** Provide memory information and recommend non-default data type changes.\n", "\n", "### What I completed\n", "- I measured total dataframe memory and identified safe dtype optimization recommendations.\n", "\n", "### Results Capture\n", "- Total memory usage: `2,263,919` bytes (`2.159 MB`).\n", "- Recommended `category` candidates: `health`, `adl`, `region`, `gender`, `married`, `employed`, `insurance`, `medicaid`.\n", "\n", "### Code evidence\n", "- The next cell contains the memory computation and candidate dtype recommendation output." ] }, { "cell_type": "code", "execution_count": 9, "id": "4c0b1deb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total memory (bytes): 2263919\n", "Total memory (MB): 2.159\n", "Recommended category columns: ['health', 'adl', 'region', 'gender', 'married', 'employed', 'insurance', 'medicaid']\n" ] } ], "source": [ "# Memory usage\n", "mem = df.memory_usage(deep=True).sum()\n", "print(\"Total memory (bytes):\", mem)\n", "print(\"Total memory (MB):\", round(mem / (1024**2), 3))\n", "candidate_category = [c for c in [\"health\",\"adl\",\"region\",\"gender\",\"married\",\"employed\",\"insurance\",\"medicaid\"] if c in df.columns]\n", "print(\"Recommended category columns:\", candidate_category)\n" ] }, { "cell_type": "markdown", "id": "4406279a", "metadata": {}, "source": [ "## C1-T9 — Export cleaned CSV for next capstone\n", "**PDF requirement:** Recommend dataframe changes and export cleaned dataframe as NSMES1988new.csv.\n", "\n", "### What I completed\n", "- I applied safe structural cleaning by removing the index-like column and exporting the cleaned dataframe.\n", "\n", "### Results Capture\n", "- Change applied: dropped `Unnamed: 0`.\n", "- Export file: `outputs/NSMES1988new.csv`; resulting shape: `(4406, 18)`.\n", "\n", "### Artifacts\n", "- `outputs/NSMES1988new.csv`\n", "\n", "### Code evidence\n", "- The next cell contains the exact cleaning and export code I ran." ] }, { "cell_type": "code", "execution_count": 10, "id": "87f28deb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dropped column: Unnamed: 0\n", "Saved: c:\\DEV_Projects\\SIMPLILEARN\\CAPSTONE_Applied_Data_Science_with _Python\\Incremental_Capstone\\Capstone 1\\outputs\\NSMES1988new.csv\n", "Shape: (4406, 18)\n" ] } ], "source": [ "# Apply safe cleaning for capstone handoff\n", "df_clean = df.copy()\n", "\n", "if \"Unnamed: 0\" in df_clean.columns:\n", " df_clean = df_clean.drop(columns=[\"Unnamed: 0\"])\n", " print(\"Dropped column: Unnamed: 0\")\n", "\n", "out_csv = BASE_DIR / \"outputs\" / \"NSMES1988new.csv\"\n", "out_csv.parent.mkdir(parents=True, exist_ok=True)\n", "\n", "df_clean.to_csv(out_csv, index=False)\n", "print(\"Saved:\", out_csv)\n", "print(\"Shape:\", df_clean.shape)" ] }, { "cell_type": "markdown", "id": "ebf73468", "metadata": {}, "source": [ "## Final section — Conclusions (required)\n", "- I completed the full Capstone 1 workflow for import, profiling, cleaning checks, and documentation.\n", "- I confirmed the dataset is complete (no missing values) and documented age/income interpretation and ranges.\n", "- I produced required artifacts: `outputs/NSMES1988.json` and `outputs/NSMES1988new.csv`.\n", "- I updated `WORK_SUMMARY.md` with evidence and marked all tasks complete.\n" ] }, { "cell_type": "code", "execution_count": 11, "id": "3063249d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Capstone 1 completed: C1-T4 to C1-T9\n", "Artifacts: outputs/NSMES1988.json and outputs/NSMES1988new.csv\n" ] } ], "source": [ "print(\"Capstone 1 completed: C1-T4 to C1-T9\")\n", "print(\"Artifacts: outputs/NSMES1988.json and outputs/NSMES1988new.csv\")\n" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.0" } }, "nbformat": 4, "nbformat_minor": 5 }